The DOI matches its DataCite record; 12 of the 12 references it cites were found in a registry or an index.
Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
Abstract
While the history of machine learning so far largely encompasses a series of problems posed by researchers and algorithms that learn their solutions, an important question is whether the problems themselves can be generated by the algorithm at the same time as they are being solved. Such a process would in effect build its own diverse and expanding curricula, and the solutions to problems at various stages would become stepping stones towards solving even more challenging problems later in the process. The Paired Open-Ended Trailblazer (POET) algorithm introduced in this paper does just that: it pairs the generation of environmental challenges and the optimization of agents to solve those challenges. It simultaneously explores many different paths through the space of possible problems and solutions and, critically, allows these stepping-stone solutions to transfer between problems if better, catalyzing innovation. The term open-ended signifies the intriguing potential for algorithms like POET to continue to create novel and increasingly complex capabilities without bound. Our results show that POET produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges, many of which cannot be solved by direct optimization alone, or even through a direct-path curriculum-building control algorithm introduced to highlight the critical role of open-endedness in solving ambitious challenges. The ability to transfer solutions from one environment to another proves essential to unlocking the full potential of the system as a whole, demonstrating the unpredictable nature of fortuitous stepping stones. We hope that POET will inspire a new push towards open-ended discovery across many domains, where algorithms like POET can blaze a trail through their interesting possible manifestations and solutions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“This kind of incremental curriculum is intuitive and variants of it appear in the literature when a task is too hard to learn directly [40, 67–70].”
BackgroundA general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018) · Science · doi:10.1126/science.aar6404 · Cited by 3,562 · cited 1×
“The ancient game of Go has been a challenge in AI for decades, and has been the subject of much recent research: AlphaGo and its variants can now reliably beat the world’s best human professional Go players [13–15].”
BackgroundRobots that can adapt like animals
Cully, A., Clune, J., Tarapore, D., & Mouret, J.-B. (2015) · Nature · doi:10.1038/nature14422 · Cited by 979
Self-citationQuality Diversity: A New Frontier for Evolutionary Computation
Pugh, J. K., Soros, L. B., & Stanley, K. O. (2016) · Frontiers in Robotics and AI · doi:10.3389/frobt.2016.00040 · Cited by 514
Self-citation
Cited by
Sorted by citations
Learning quadrupedal locomotion over challenging terrain
Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., & Hutter, M. (2020) · Science Robotics · doi:10.1126/scirobotics.abc5986 · Cited by 1,156
Unity: A General Platform for Intelligent Agents
Juliani, A., Berges, V.-P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., Mattar, M., et al. (2018) · arXiv · doi:10.48550/arxiv.1809.02627 · Cited by 626
“It has been argued in recent work that generating effective and diverse learning environments (often as a co-evolutionary process involving agent and environment) is a critical component for developing artificial general intelligence (Wang et al., 2019; Clune, 2019).”
BackgroundEmergent Tool Use From Multi-Agent Autocurricula
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019) · arXiv · doi:10.48550/arxiv.1909.07528 · Cited by 424
“[29] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley.”
BackgroundTowards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224
Voyager: An Open-Ended Embodied Agent with Large Language Models
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023) · arXiv · doi:10.48550/arxiv.2305.16291 · Cited by 222
“[42] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O.”
BackgroundA Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193
“Wang et al. (2019) introduced Paired Open-Ended Trailblazer (POET), a method for jointly evolving context-MDPs and policies which solve those MDPs, aiming for a policy that can solve a wide variety of context-MDPs.”
MethodTriguero, I., Molina, D., Poyatos, J., Del Ser, J., & Herrera, F. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102135 · Cited by 90
Clune, J. (2019) · arXiv · doi:10.48550/arxiv.1905.10985 · Cited by 82
Self-citationTeacher algorithms for curriculum learning of Deep RL in continuously parameterized environments
Portelas, R., Colas, C., Hofmann, K., & Oudeyer, P.-Y. (2019) · arXiv · doi:10.48550/arxiv.1910.07224 · Cited by 59
Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
Ha, H., Florence, P., & Song, S. (2023) · arXiv · doi:10.48550/arxiv.2307.14535 · Cited by 24
“Towards unsupervised exploration, prior works have also investigated evolving environments [26, 27] and embodiments [28], automatic task generation [29], leveraging language guidance [30, 31] and world-model error [32], but have not been demonstrated to scale to 6 DoF robotic skill learning.”
BackgroundReplay-Guided Adversarial Environment Design
Jiang, M., Dennis, M., Parker-Holder, J., Foerster, J., Grefenstette, E., & Rocktäschel, T. (2021) · arXiv · doi:10.48550/arxiv.2110.02439 · Cited by 21
Human-Timescale Adaptation in an Open-Ended Task Space
Adaptive Agent Team, Bauer, J., Baumli, K., Baveja, S., Behbahani, F., Bhoopchand, A., Bradley-Schmieg, N., Chang, M., Clay, N., Collister, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.07608 · Cited by 19
“Recent works have demonstrated the effectiveness of agent-environment co-adaptation with a distribution of tasks (Wang et al., 2019; Akkaya et al., 2019; Wang et al., 2020a; Parker-Holder et al., 2022).”
BackgroundAbsolute Zero: Reinforced Self-play Reasoning with Zero Data
Zhao, A., Wu, Y., Yue, Y., Wu, T., Xu, Q., Yue, Y., Lin, M., Wang, S., Wu, Q., Zheng, Z., et al. (2025) · arXiv · Cited by 13
Multi-Agent Risks from Advanced AI
Hammond, L., Chan, A., Clifton, J., Hoelscher-Obermaier, J., Khan, A., McLean, E., Smith, C., Barfuss, W., Foerster, J., Gavenčiak, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.14143 · Cited by 305
Self-citationDarwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
Zhang, J., Hu, S., Lu, C., Lange, R., & Clune, J. (2025) · arXiv · doi:10.48550/arxiv.2505.22954 · Cited by 344
Self-citationCraftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning
Matthews, M., Beukman, M., Ellis, B., Samvelyan, M., Jackson, M., Coward, S., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2402.16801 · Cited by 4
“In other words, Craftax assesses deep exploration whereas XLand-Minigrid assesses wide generalisation.”
BackgroundMAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning
Samvelyan, M., Khan, A., Dennis, M., Jiang, M., Parker-Holder, J., Foerster, J., Raileanu, R., & Rocktäschel, T. (2023) · arXiv · doi:10.48550/arxiv.2303.03376 · Cited by 8
DarwinX: Evolving Agent Harnesses Through Natural Selection
Zhang, Y., Dai, Y., Tan, J., Yang, L., Mullur, R., Hoang, T., Hu, Z., Zhu, J., Mui, P., Savarese, S., et al. (2026) · arXiv · doi:10.48550/arxiv.2608.07545 · Cited by 1
“An adjacent line evolves task-solution programs rather than agents [25, 35, 38], building on quality-diversity and open-ended search [13, 24, 33, 46] and contributing the archive and cascade machinery agent evolution inherits, but presuming a deterministic fitness signal that agentic benchmarks do…”
Generalization through Diversity: Improving Unsupervised Environment Design
Li, W., Varakantham, P., & Li, D. (2023) · arXiv · Cited by 3
“For instance, in Bipedal-Walker, a complex yet well-parameterized UPOMDP introduced by [Wang et al. , 2019b; Parker-Holder et al. , 2022], has multiple free parameters controlling the terrain.”
BackgroundMethodInfluentialGeneralization through Diversity: Improving Unsupervised Environment Design
Li, W., Varakantham, P., & Li, D. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/601 · Cited by 3
“For instance, in Bipedal-Walker, a complex yet well-parameterized UPOMDP introduced by [Wang et al. , 2019b; Parker-Holder et al. , 2022], has multiple free parameters controlling the terrain.”
BackgroundMethodInfluentialKinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks
Matthews, M., Beukman, M., Lu, C., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2410.23208 · Cited by 1
“While these methods hold the promise of generating novel and useful levels in an open-ended manner, the environments used in their experiments are often very constrained in what they can rep-9 resent (Wang et al., 2019; Dennis et al., 2020; Jiang et al., 2021b;a; Parker-Holder et al., 2022).”
Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairs
Furelos-Blanco, D., Pert, C., Kelbel, F., Spies, A. F., Russo, A., & Dennis, M. (2025) · arXiv
“POET (Wang et al. 2019) trains populations of specialist agents using evolutionary strategies, while ATLAS trains general agents from random and co-evolved task-level pairs.”
Improving Environment Novelty Quantification for Effective Unsupervised Environment Design
Teoh, J., Li, W., & Varakantham, P. (2025) · arXiv
“We also evaluated the CENIE-augmented algorithms in the BipedalWalker domain [56, 37], which is a partially observable continuous domain with dense rewards.”
InfluentialImproving Environment Novelty Quantification for Effective Unsupervised Environment Design
Teoh, J., Li, W., & Varakantham, P. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-4300
Metadata
- Authors
- Wang, R., Lehman, J., Clune, J., & Stanley, K. O.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- arXiv
- 1901.01753
- References
- 12
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
