The DOI matches its DataCite record; 6 of the 6 references it cites were found in a registry or an index.
Quantifying Generalization in Reinforcement Learning
Abstract
In this paper, we investigate the problem of overfitting in deep reinforcement learning. Among the most common benchmarks in RL, it is customary to use the same environments for both training and testing. This practice offers relatively little insight into an agent's ability to generalize. We address this issue by using procedurally generated environments to construct distinct training and test sets. Most notably, we introduce a new environment called CoinRun, designed as a benchmark for generalization in RL. Using CoinRun, we find that agents overfit to surprisingly large training sets. We then show that deeper convolutional architectures improve generalization, as do methods traditionally found in supervised learning, including L2 regularization, dropout, data augmentation and batch normalization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DRL agents often fail to generalize well to environments only slightly different from those encountered during training [21, 63].”
From Improving Environment Novelty Quantification for Effective Unsupervised Environment Design (2024) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Self-citationPlaying Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5699 · Cited by 359
Assessing Generalization in Deep Reinforcement Learning
Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1810.12282 · Cited by 114
Generalization and Regularization in DQN
Farebrother, J., Machado, M. C., & Bowling, M. (2018) · arXiv · doi:10.48550/arxiv.1810.00123 · Cited by 92 · cited 2×
“(Farebrother et al., 2018) also address this problem, accurately recognizing that conflating train and test environments has contributed to the lack of regularization in deep RL.”
Background
Research works printed but not linked: 15
- M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation plat- form for general agents. CoRR, abs/1207.4708, 2012. · arXiv
- E. D. Cubuk, B. Zoph, D. Mané, V. Vasudevan, and Q. V. Le. Autoaugment: Learning augmentation policies from data. CoRR, abs/1805.09501, 2018. URL http:// arxiv.org/abs/1805.09501. · arXiv
- T. Devries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. CoRR, abs/1708.04552, 2017. URL http://arxiv.org/ abs/1708.04552. · arXiv
- L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA: scalable dis- tributed deep-rl with importance weighted actor-learner architectures. CoRR, abs/1802.01561, 2018. · arXiv
- M. J. Hausknecht and P. Stone. The impact of deter- minism on learning atari 2600 games. In Learning for General Competency in Video Games, Papers from the 2015 AAAI Workshop, Austin, Texas, USA, January 26, 2015., 2015. URL http://aaai.org/ocs/ index.php/WS/AAAIW15/paper/view/9564.
- S. Hochreiter and J. Schmidhuber. Long short-term mem- ory. Neural Computation, 9(8):1735-1780, 1997.
- S. Ioffe and C. Szegedy. Batch normalization: Accelerat- ing deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015. · arXiv
- N. Justesen, R. R. Torrado, P. Bontrager, A. Khalifa, J. To- gelius, and S. Risi. Illuminating generalization in deep reinforcement learning through procedural level genera- tion. CoRR, abs/1806.10729, 2018. · arXiv
- J. B. Kruskal. On the shortest spanning subtree of a graph and the traveling salesman problem. In Proceedings of the American Mathematical Society, 7, pages 48-50, 1956.
- P. Luo, X. Wang, W. Shao, and Z. Peng. Towards under- standing regularization in batch normalization. CoRR, abs/1809.00846, 2018. URL http://arxiv.org/ abs/1809.00846. · arXiv
- V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fid- jeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529- 533, 2015.
- A. Nichol, V. Pfau, C. Hesse, O. Klimov, and J. Schulman. Gotta learn fast: A new benchmark for generalization in RL. CoRR, abs/1804.03720, 2018. URL http:// arxiv.org/abs/1804.03720. · arXiv
- J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. CoRR, abs/1703.06907, 2017. URL http://arxiv. org/abs/1703.06907. · arXiv
- A. Zhang, N. Ballas, and J. Pineau. A dissection of over- fitting and generalization in continuous reinforcement learning. CoRR, abs/1806.07937, 2018a. · arXiv
- C. Zhang, O. Vinyals, R. Munos, and S. Bengio. A study on overfitting in deep reinforcement learning. CoRR, abs/1804.06893, 2018b. URL http://arxiv.org/ abs/1804.06893. · arXiv
Cited by
Sorted by citations
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.06680 · Cited by 1,065
Self-citationMetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
Li, Q., Peng, Z., Feng, L., Zhang, Q., Xue, Z., & Zhou, B. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3190471 · Cited by 246
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193
“Cobbe et al. (2019) introduced CoinRun, and evaluated standard supervised learning regularisation techniques on the benchmark.”
BackgroundMethodInfluentialHierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129
Assessing Generalization in Deep Reinforcement Learning
Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1810.12282 · Cited by 114
Generalization in Reinforcement Learning by Soft Data Augmentation
Hansen, N., & Wang, X. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561103 · Cited by 111
Vision-Language Navigation with Random Environmental Mixup
Liu, C., Zhu, F., Chang, X., Liang, X., Ge, Z., & Shen, Y.-D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00167 · Cited by 80
Imitating Latent Policies from Observation
Edwards, A. D., Sahni, H., Schroecker, Y., & Isbell, C. L. (2018) · arXiv · doi:10.48550/arxiv.1805.07914 · Cited by 79
Observational Overfitting in Reinforcement Learning
Song, X., Jiang, Y., Tu, S., Du, Y., & Neyshabur, B. (2019) · arXiv · doi:10.48550/arxiv.1912.02975 · Cited by 48
Human-Timescale Adaptation in an Open-Ended Task Space
Adaptive Agent Team, Bauer, J., Baumli, K., Baveja, S., Behbahani, F., Bhoopchand, A., Bradley-Schmieg, N., Chang, M., Clay, N., Collister, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.07608 · Cited by 19
“PCG has been studied for many years in the games community (Togelius and Schmidhuber, 2008; Risi and Togelius, 2020) and more recently has been used to create testbeds for RL agents (Justesen et al., 2018; Cobbe et al., 2018; Raileanu and Rockt¨aschel, 2020).”
MethodInterference and Generalization in Temporal Difference Learning
Bengio, E., Pineau, J., & Precup, D. (2020) · arXiv · doi:10.48550/arxiv.2003.06350 · Cited by 14
Generalization to New Actions in Reinforcement Learning
Jain, A., Szot, A., & Lim, J. J. (2020) · arXiv · doi:10.48550/arxiv.2011.01928 · Cited by 14
“Previously, state-space generalization has been used to transfer policies to new environments (Cobbe et al., 2018; Nichol et al., 2018; Packer et al., 2018), agent morphologies (Wang et al., 2018; Sanchez-Gonzalez et al., 2018; Pathak et al., 2019), and visual inputs for manipulation of unseen…”
BackgroundMethodMeasuring Visual Generalization in Continuous Control from Pixels
Grigsby, J., & Qi, Y. (2020) · arXiv · doi:10.48550/arxiv.2010.06740 · Cited by 10
Generalization through Diversity: Improving Unsupervised Environment Design
Li, W., Varakantham, P., & Li, D. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/601 · Cited by 3
“However, DRL agents have been proven to be brittle and often fail to transfer well to environments only slightly different from those encountered during training [Zhang et al. , 2018; Cobbe et al. , 2019], e.g., changing size of the maze, background in a game, positions of obstacles.”
BackgroundGeneralization through Diversity: Improving Unsupervised Environment Design
Li, W., Varakantham, P., & Li, D. (2023) · arXiv · Cited by 3
“However, DRL agents have been proven to be brittle and often fail to transfer well to environments only slightly different from those encountered during training [Zhang et al. , 2018; Cobbe et al. , 2019], e.g., changing size of the maze, background in a game, positions of obstacles.”
BackgroundA Study of Global and Episodic Bonuses for Exploration in Contextual MDPs
Henaff, M., Jiang, M., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2306.03236 · Cited by 1
“…singleton MDPs exhibit poor generalization, and that even minor changes to the environment can cause substantial degradation in agent performance (Justesen et al., 2018; Zhang et al., 2018a;b; Packer et al., 2018; Farebrother et al., 2018; Cobbe et al., 2019; Song et al., 2020; Kirk et al., 2021a).”
BackgroundKinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks
Matthews, M., Beukman, M., Lu, C., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2410.23208 · Cited by 1
“…with some notable exceptions (Team et al., 2021; 2023), most RL environments represent a narrow and homogeneous set of scenarios (Todorov et al., 2012; Bellemare et al., 2013; Brockman et al., 2016; Cobbe et al., 2019), limiting the generalisation ability of the trained agents (Kirk et al., 2023).”
Improving Environment Novelty Quantification for Effective Unsupervised Environment Design
Teoh, J., Li, W., & Varakantham, P. (2025) · arXiv
“DRL agents often fail to generalize well to environments only slightly different from those encountered during training [21, 63].”
Improving Environment Novelty Quantification for Effective Unsupervised Environment Design
Teoh, J., Li, W., & Varakantham, P. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-4300
ProcgenOOD: a benchmark for out-of-distribution generalization in reinforcement learning
Corder, K., Park, S. J., & Vindiola, M. M. (2026) · Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications VIII · doi:10.1117/12.3093362
Metadata
- Authors
- Cobbe, K., Klimov, O., Hesse, C., Kim, T., & Schulman, J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- arXiv
- 1812.02341
- References
- 6
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
