The DOI matches its DataCite record; 6 of the 6 references it cites were found in a registry or an index.
Quantifying Generalization in Reinforcement Learning
Abstract
In this paper, we investigate the problem of overfitting in deep reinforcement learning. Among the most common benchmarks in RL, it is customary to use the same environments for both training and testing. This practice offers relatively little insight into an agent's ability to generalize. We address this issue by using procedurally generated environments to construct distinct training and test sets. Most notably, we introduce a new environment called CoinRun, designed as a benchmark for generalization in RL. Using CoinRun, we find that agents overfit to surprisingly large training sets. We then show that deeper convolutional architectures improve generalization, as do methods traditionally found in supervised learning, including L2 regularization, dropout, data augmentation and batch normalization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“DRL agents often fail to generalize well to environments only slightly different from those encountered during training [21, 63].”
From Improving Environment Novelty Quantification for Effective Unsupervised Environment Design (2024) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Self-citationPlaying Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5699 · Cited by 359
Assessing Generalization in Deep Reinforcement Learning
Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1810.12282 · Cited by 114
Generalization and Regularization in DQN
Farebrother, J., Machado, M. C., & Bowling, M. (2018) · arXiv · doi:10.48550/arxiv.1810.00123 · Cited by 92 · cited 2×
“(Farebrother et al., 2018) also address this problem, accurately recognizing that conflating train and test environments has contributed to the lack of regularization in deep RL.”
Background
Flagged first, then by citations
Research works printed but not linked: 15
- M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation plat- form for general agents. CoRR, abs/1207.4708, 2012. · arXiv
- E. D. Cubuk, B. Zoph, D. Mané, V. Vasudevan, and Q. V. Le. Autoaugment: Learning augmentation policies from data. CoRR, abs/1805.09501, 2018. URL http:// arxiv.org/abs/1805.09501. · arXiv
- T. Devries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. CoRR, abs/1708.04552, 2017. URL http://arxiv.org/ abs/1708.04552. · arXiv
- L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA: scalable dis- tributed deep-rl with importance weighted actor-learner architectures. CoRR, abs/1802.01561, 2018. · arXiv
- M. J. Hausknecht and P. Stone. The impact of deter- minism on learning atari 2600 games. In Learning for General Competency in Video Games, Papers from the 2015 AAAI Workshop, Austin, Texas, USA, January 26, 2015., 2015. URL http://aaai.org/ocs/ index.php/WS/AAAIW15/paper/view/9564.
- S. Hochreiter and J. Schmidhuber. Long short-term mem- ory. Neural Computation, 9(8):1735-1780, 1997.
- S. Ioffe and C. Szegedy. Batch normalization: Accelerat- ing deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015. · arXiv
- N. Justesen, R. R. Torrado, P. Bontrager, A. Khalifa, J. To- gelius, and S. Risi. Illuminating generalization in deep reinforcement learning through procedural level genera- tion. CoRR, abs/1806.10729, 2018. · arXiv
- J. B. Kruskal. On the shortest spanning subtree of a graph and the traveling salesman problem. In Proceedings of the American Mathematical Society, 7, pages 48-50, 1956.
- P. Luo, X. Wang, W. Shao, and Z. Peng. Towards under- standing regularization in batch normalization. CoRR, abs/1809.00846, 2018. URL http://arxiv.org/ abs/1809.00846. · arXiv
- V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fid- jeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529- 533, 2015.
- A. Nichol, V. Pfau, C. Hesse, O. Klimov, and J. Schulman. Gotta learn fast: A new benchmark for generalization in RL. CoRR, abs/1804.03720, 2018. URL http:// arxiv.org/abs/1804.03720. · arXiv
- J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. CoRR, abs/1703.06907, 2017. URL http://arxiv. org/abs/1703.06907. · arXiv
- A. Zhang, N. Ballas, and J. Pineau. A dissection of over- fitting and generalization in continuous reinforcement learning. CoRR, abs/1806.07937, 2018a. · arXiv
- C. Zhang, O. Vinyals, R. Munos, and S. Bengio. A study on overfitting in deep reinforcement learning. CoRR, abs/1804.06893, 2018b. URL http://arxiv.org/ abs/1804.06893. · arXiv
Cited by
Sorted by citations
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.06680 · Cited by 1,065
Self-citationMetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
Li, Q., Peng, Z., Feng, L., Zhang, Q., Xue, Z., & Zhou, B. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3190471 · Cited by 246
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193
“Cobbe et al. (2019) introduced CoinRun, and evaluated standard supervised learning regularisation techniques on the benchmark.”
BackgroundMethodInfluentialHierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129
Metadata
- Authors
- Cobbe, K., Klimov, O., Hesse, C., Kim, T., & Schulman, J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- arXiv
- 1812.02341
- References
- 6
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
