Leveraging Procedural Generation to Benchmark Reinforcement Learning
Open accessAbstract
We introduce Procgen Benchmark, a suite of 16 procedurally generated game-like environments designed to benchmark both sample efficiency and generalization in reinforcement learning. We believe that the community will benefit from increased access to high quality training environments, and we provide detailed experimental protocols for using this benchmark. We empirically demonstrate that diverse environment distributions are essential to adequately train and evaluate RL agents, thereby motivating the extensive use of procedural content generation. We then use this benchmark to investigate the effects of scaling model size, finding that larger models significantly improve both sample efficiency and generalization.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Even minor changes to the environment can cause substantial degradation in agent performance.
“even minor changes to the environment can cause substantial degradation in agent performance”
From A Study of Global and Episodic Bonuses for Exploration in Contextual MDPs (2023) · cites this paper as Cobbe et al., 2019
Section 3.2 (p. 4) reports a vast gap between training and test performance when agents trained on a fixed level sequence are tested on randomly chosen sequences. Section 3.1 (p. 4) also reports strong overfitting to small training sets. These findings support substantial performance degradation on changed levels, but the paper does not establish that the changes are minor. (LLM, conf=0.84)
Experimental — accuracy not yet measuredWithin the budget of 50M frames, Dreamer matches the tuned expert algorithm PPG and outperforms Rainbow on ProcGen.
“Within the budget of 50M frames, Dreamer matches the tuned expert algorithm PPG 37 and outperforms Rainbow 42,40 .”
From Mastering Diverse Domains through World Models (2023) · cites this paper as [42]
Section 5 (p. 6) evaluates Rainbow on Procgen, supporting the citation as a source for that benchmark comparison. It does not compare Rainbow with Dreamer, evaluate PPG, or report the claimed result within 50M frames. (LLM, conf=0.84)
Experimental — accuracy not yet measuredDeep Reinforcement Learning agents have been proven brittle and often fail to transfer well to environments only slightly different from those encountered during training.
“However, DRL agents have been proven to be brittle and often fail to transfer well to environments only slightly different from those encountered during training [Zhang et al. , 2018; Cobbe et al. , 2019], e.g., changing size of the maze, background in a game, positions of obstacles.”
From Generalization through Diversity: Improving Unsupervised Environment Design (2023) · cites this paper as Cobbe et al. , 2019
We compare ProcGen at 50M steps due to computational cost, as there is no action repeat.
“Prior work compares at different step budgets 42,37 and we compare at 50M steps due to computational cost, as there is no action repeat.”
From Mastering Diverse Domains through World Models (2023) · cites this paper as [42]
All the regularisation techniques separately improve performance, and the combination improves performance further, although the gains of the combination is minimal over the individual methods, implying that many of these methods address similar causes of worse generalisation.
“All the techniques separately improve performance, and the combination improves performance further, although the gains of the combination is minimal over the individual methods, implying that many of these methods address similar causes of worse generalisation.”
From A Survey of Zero-shot Generalisation in Deep Reinforcement Learning (2023) · cites this paper as Cobbe et al., 2019
Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 25 · Cited by 196
Research works printed but not linked: 17
- M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, jun 2013.
- K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman. Quantifying generalization in reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , pages 1282-1289, 2019. URL http://proceedings.mlr.press/ v97/cobbe19a.html .
- L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. CoRR , abs/1802.01561, 2018. · arXiv
- X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256, 2010.
- M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018.
- S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation , 9(8):1735-1780, 1997.
- L. Johnson, G. N. Yannakakis, and J. Togelius. Cellular automata for real-time generation of infinite cave levels. In Proceedings of the 2010 Workshop on Procedural Content Generation in Games , page 10. ACM, 2010.
- A. Juliani, A. Khalifa, V.-P. Berges, J. Harper, H. Henry, A. Crespi, J. Togelius, and D. Lange. Obstacle tower: A generalization challenge in vision, control, and planning. arXiv preprint arXiv:1902.01378 , 2019. · arXiv
- N. Justesen, R. R. Torrado, P. Bontrager, A. Khalifa, J. Togelius, and S. Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. CoRR , abs/1806.10729, 2018. · arXiv
- J. B. Kruskal. On the shortest spanning subtree of a graph and the traveling salesman problem. In Proceedings of the American Mathematical Society, 7 , pages 48-50, 1956.
- K. Lee, K. Lee, J. Shin, and H. Lee. A simple randomization technique for generalization in deep reinforcement learning. arXiv preprint arXiv:1910.05396 , 2019. · arXiv
- V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529533, 2015.
- A. Nichol, V. Pfau, C. Hesse, O. Klimov, and J. Schulman. Gotta learn fast: A new benchmark for generalization in RL. CoRR , abs/1804.03720, 2018. URL http:// arxiv.org/abs/1804.03720 . · arXiv
- D. Perez-Liebana, J. Liu, A. Khalifa, R. D. Gaina, J. Togelius, and S. M. Lucas. General video game ai: a multi-track framework for evaluating agents, games and content generation algorithms. arXiv preprint arXiv:1802.10363 , 2018. · arXiv
- A. Zhang, N. Ballas, and J. Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. CoRR , abs/1806.07937, 2018a. · arXiv
- A. Zhang, Y. Wu, and J. Pineau. Natural environment benchmarks for reinforcement learning. arXiv preprint arXiv:1811.06032 , 2018b. · arXiv
- C. Zhang, O. Vinyals, R. Munos, and S. Bengio. A study on overfitting in deep reinforcement learning. CoRR , abs/1804.06893, 2018c. URL http://arxiv.org/ abs/1804.06893 . · arXiv
- Authors
- Cobbe, K., Hesse, C., Hilton, J., & Schulman, J.
- Affiliations
- OpenAI (United States) (US)
- ORCID
- Hilton, Jacob
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1912.01588
- OpenAlex
- W2994073215
- Cited by
- 196
- References
- 25
- Citations by year
- 2025: 10 · 2024: 31 · 2023: 38 · 2022: 49 · 2021: 37 · 2020: 30 · 2018: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- procedural content generation, sample efficiency, generalization, model size scaling, training environment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
