A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
Open accessAbstract
The risks and perils of overfitting in machine learning are well known. However most of the treatment of this, including diagnostic tools and remedies, was developed for the supervised learning case. In this work, we aim to offer new perspectives on the characterization and prevention of overfitting in deep Reinforcement Learning (RL) methods, with a particular focus on continuous domains. We examine several aspects, such as how to define and diagnose overfitting in MDPs, and how to reduce risks by injecting sufficient training diversity. This work complements recent findings on the brittleness of deep RL methods and offers practical observations for RL researchers and practitioners.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reinforcement learning is extremely sample inefficient and prone to overfitting, sometimes failing to generalize to even subtle variations in environmental dynamics or goals.
“Reinforcement learning (RL) is known to be extremely sample inefficient and prone to overfitting, sometimes failing to generalize to even subtle variations in environmental dynamics or goals”
From XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX (2024) · cites this paper as Zhang et al., 2018
The Introduction (p. 2) reports that deep RL algorithms can overfit, and the Conclusion (p. 9) calls for a drastic improvement in RL sample complexity. Section 4.1 (p. 4) shows overfitting in a task with a changed goal, but the paper does not demonstrate failure under subtle changes in transition dynamics. The claim also generalizes findings about deep RL methods to RL as a whole. (LLM, conf=0.84)
Experimental — accuracy not yet measuredDeep Reinforcement Learning agents have been proven brittle and often fail to transfer well to environments only slightly different from those encountered during training.
“However, DRL agents have been proven to be brittle and often fail to transfer well to environments only slightly different from those encountered during training [Zhang et al. , 2018; Cobbe et al. , 2019], e.g., changing size of the maze, background in a game, positions of obstacles.”
From Generalization through Diversity: Improving Unsupervised Environment Design (2023) · cites this paper as Zhang et al. , 2018
Section 5 (p. 8) reports poorer transfer performance when noise is added to observations and says shifts in the initial-state distribution make generalization harder. But the Introduction (p. 2) also says deep RL generalizes well in simulated environments with enough training diversity. The source supports brittleness under some changed conditions, not the unqualified suggestion that agents often fail after slight environmental changes; it does not test the maze, background, or obstacle examples. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685 · see citing works
Self-citationMachado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5699 · Cited by 359 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [2]2. Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. CoRR , abs/1612.03801, 2016. - R. Bellman. A Markovian Decision Process. Journal of Mathematics and Mechanics , 1957. · arXiv
- [4]4. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. - V. Francois-Lavet, D. Ernst, and R. Fonteneau. On overfitting and asymptotic bias in batch reinforcement learning with partial observability. ArXiv e-prints , September 2017.
- [12]12. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 02 2015. URL http://dx.doi.org/10.1038/nature14236 . · DOI · Paper page
- [13]13. Susan A. Murphy. A generalization error for q-learning. J. Mach. Learn. Res. , 6:1073-1097, December 2005. ISSN 1532-4435. URL http://dl.acm.org/citation.cfm?id=1046920. 1088709 . - J. Oh, S. Singh, H. Lee, and P. Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In ICML , 2017.
- [15]15. Marc Ponsen, Matthew E. Taylor, and Karl Tuyls. Abstraction and generalization in reinforcement learning: A summary and framework. In Matthew E. Taylor and Karl Tuyls, editors, Adaptive and Learning Agents , pages 1-32, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg. ISBN 978-3-642-11814-2. - A. Rajeswaran, K. Lowrey, E. Todorov, and S. Kakade. Towards Generalization and Simplicity in Continuous Control. ArXiv e-prints , March 2017. - John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017. - Richard S. Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Advances in Neural Information Processing Systems 8 , pages 1038-1044. MIT Press, 1996. - M. Taylor and P. Stone. Transfer learning for reinforcement learning domains: A survey. JMLR , 2009. - Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS , pages 5026-5033. IEEE, 2012. ISBN 978-1-4673-1737-5. URL http://dblp.uni-trier. de/db/conf/iros/iros2012.html#TodorovET12 . - Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double qlearning. CoRR , abs/1509.06461, 2015. - S. Whiteson, B. Tanner, M. E. Taylor, and P. Stone. Protecting against evaluation overfitting in empirical reinforcement learning. In 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL) , pages 120-127, April 2011. doi: 10.1109/ADPRL.2011. 5967363. - C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. ArXiv e-prints , November 2016. - C. Zhang, O. Vinyals, R. Munos, and S. Bengio. A Study on Overfitting in Deep Reinforcement Learning. ArXiv e-prints , April 2018. · DOI
Cited by
Sorted by citations
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193 · see citing works
“Wang et al. (2020, Mixreg) adapts mixup (Zhang et al., 2018) to the RL setting, which encourages the policy to be linear in its outputs with respect to mixtures of possible inputs.”
BackgroundInfluentialSelf-citationAssessing Generalization in Deep Reinforcement Learning
Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1810.12282 · Cited by 114 · see citing works
Replay-Guided Adversarial Environment Design
Jiang, M., Dennis, M., Parker-Holder, J., Foerster, J., Grefenstette, E., & Rocktäschel, T. (2021) · arXiv · doi:10.48550/arxiv.2110.02439 · Cited by 21 · see citing works
Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
Igl, M., Farquhar, G., Luketina, J., Boehmer, W., & Whiteson, S. (2020) · arXiv · doi:10.48550/arxiv.2006.05826 · Cited by 17 · see citing works
Interference and Generalization in Temporal Difference Learning
Bengio, E., Pineau, J., & Precup, D. (2020) · arXiv · doi:10.48550/arxiv.2003.06350 · Cited by 14 · see citing works
Self-citationRobust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation
Zhao, C., & Hospedales, T. (2020) · arXiv · doi:10.48550/arxiv.2012.04839 · Cited by 11 · see citing works
XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX
Nikulin, A., Kurenkov, V., Zisman, I., Agarkov, A., Sinii, V., & Kolesnikov, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1390 · Cited by 3 · see citing works
“Reinforcement learning (RL) is known to be extremely sample inefficient and prone to overfitting, sometimes failing to generalize to even subtle variations in environmental dynamics or goals (Ra-jeswaran et al., 2017; Zhang et al., 2018; Henderson et al., 2018; Alver & Precup, 2020).”
BackgroundGeneralization through Diversity: Improving Unsupervised Environment Design
Li, W., Varakantham, P., & Li, D. (2023) · arXiv · Cited by 3 · see citing works
“However, DRL agents have been proven to be brittle and often fail to transfer well to environments only slightly different from those encountered during training [Zhang et al. , 2018; Cobbe et al. , 2019], e.g., changing size of the maze, background in a game, positions of obstacles.”
BackgroundGeneralization through Diversity: Improving Unsupervised Environment Design
Li, W., Varakantham, P., & Li, D. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/601 · Cited by 3 · see citing works
“However, DRL agents have been proven to be brittle and often fail to transfer well to environments only slightly different from those encountered during training [Zhang et al. , 2018; Cobbe et al. , 2019], e.g., changing size of the maze, background in a game, positions of obstacles.”
BackgroundA Study of Global and Episodic Bonuses for Exploration in Contextual MDPs
Henaff, M., Jiang, M., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2306.03236 · Cited by 1 · see citing works
“…singleton MDPs exhibit poor generalization, and that even minor changes to the environment can cause substantial degradation in agent performance (Justesen et al., 2018; Zhang et al., 2018a;b; Packer et al., 2018; Farebrother et al., 2018; Cobbe et al., 2019; Song et al., 2020; Kirk et al., 2021a).”
Background
- Authors
- Zhang, A., Ballas, N., & Pineau, J.
- ORCID
- Zhang, Amy
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1806.07937
- OpenAlex
- W2809668646
- Cited by
- 88
- References
- 4
- Citations by year
- 2025: 4 · 2024: 3 · 2023: 7 · 2022: 9 · 2021: 29 · 2020: 20 · 2019: 13 · 2018: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- overfitting, generalization, continuous reinforcement learning, deep reinforcement learning, Markov decision process, brittleness, overfitting reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
