Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
Open accessAbstract
Non-stationarity can arise in Reinforcement Learning (RL) even in stationary environments. For example, most RL algorithms collect new data throughout training, using a non-stationary behaviour policy. Due to the transience of this non-stationarity, it is often not explicitly addressed in deep RL and a single neural network is continually updated. However, we find evidence that neural networks exhibit a memory effect where these transient non-stationarities can permanently impact the latent representation and adversely affect generalisation performance. Consequently, to improve generalisation of deep RL agents, we propose Iterated Relearning (ITER). ITER augments standard RL training by repeated knowledge transfer of the current policy into a freshly initialised network, which thereby experiences less non-stationarity during training. Experimentally, we show that ITER improves performance on the challenging generalisation benchmarks ProcGen and Multiroom.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037 · see citing works
Li, Z., & Hoiem, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2773081 · Cited by 4,202 · see citing works
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 2× · see citing works
“Several possibilities have been proposed to further improve generalisation, including to provide more diverse training environments (Tobin et al., 2017), inject noise into the environment (Stulp et al., 2011; Lee et al., 2020), incorporate inductive biases in the architecture (Kansky et al., 2017),…”
BackgroundContinual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117 · see citing works
Catastrophic forgetting in connectionist networks
French, R. (1999) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(99)01294-2 · Cited by 2,311 · see citing works
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941 · see citing works
Representational geometry: integrating cognition, computation, and the brain
Kriegeskorte, N., & Kievit, R. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.06.007 · Cited by 898 · see citing works
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner ArchitecturesPreprint
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.01561 · Cited by 851 · see citing works
Born Again Neural NetworksPreprint
Furlanello, T., Lipton, Z. C., Tschannen, M., Itti, L., & Anandkumar, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04770 · Cited by 798 · see citing works
Measuring Catastrophic Forgetting in Neural Networks
Kemker, R., McClure, M., Abitino, A., Hayes, T., & Kanan, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11651 · Cited by 672 · see citing works
Multiple model-based reinforcement learning
Doya, K., Samejima, K., Katagiri, K.-I., & Kawato, M. (2002) · Neural Computation · doi:10.1162/089976602753712972 · Cited by 476 · see citing works
A Study on Overfitting in Deep Reinforcement LearningPreprint
Zhang, C., Vinyals, O., Munos, R., & Samy Bengio, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.06893 · Cited by 246 · see citing works
Generalization in Reinforcement Learning with Selective Noise Injection and Information BottleneckPreprint
Igl, M., Ciosek, K., Li, Y.-Z., Tschiatschek, S., Zhang, C., Devlin, S., & Hofmann, K. (2019) · Neural Information Processing Systems · doi:10.48550/arxiv.1910.12911 · Cited by 206 · see citing works
Self-citationLeveraging Procedural Generation to Benchmark Reinforcement Learning
Cobbe, K., Hesse, C., Hilton, J., & Schulman, J. (2019) · arXiv · doi:10.48550/arxiv.1912.01588 · Cited by 196 · see citing works
Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive PhysicsPreprint
Kansky, K., Silver, T., Mély, D. A., Eldawy, M., Lázaro-Gredilla, M., Lou, X., Dorfman, N., Sidor, S., Phoenix, S., & Dileep George, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.04317 · Cited by 176 · see citing works
Revisiting Fundamentals of Experience ReplayPreprint
Fedus, W., Ramachandran, P., Agarwal, R., Bengio, Y., Larochelle, H., Rowland, M., & Dabney, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06700 · Cited by 166 · see citing works
Self-Distillation Amplifies Regularization in Hilbert SpacePreprint
Mobahi, H., Farajtabar, M., & Bartlett, P. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.05715 · Cited by 128 · see citing works
Distral: Robust Multitask Reinforcement Learning
Teh, Y. W. (2025) · TIB Data Manager · doi:10.57702/nufubi9l · Cited by 126 · see citing works
Network Randomization: A Simple Technique for Generalization in Deep Reinforcement LearningPreprint
Lee, K., Lee, K., Shin, J., & Lee, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05396 · Cited by 97 · see citing works
A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
Zhang, A., Ballas, N., & Pineau, J. (2018) · arXiv · doi:10.48550/arxiv.1806.07937 · Cited by 88 · see citing works
Learning to grasp under uncertainty
Stulp, F., Theodorou, E. A., Buchli, J., & Stefan Schaal, A. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5979644 · Cited by 86 · see citing works
Distilling Policy DistillationPreprint
Czarnecki, W. M., Pascanu, R., Osindero, S., Jayakumar, S. M., Świrszcz, G., & Jaderberg, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02186 · Cited by 72 · see citing works
Kickstarting Deep Reinforcement Learning
Schmitt, S., Hudson, J. J., Zidek, A., Osindero, S., Doersch, C., Czarnecki, W. M., Leibo, J. Z., Kuttler, H., Zisserman, A., Simonyan, K., et al. (2018) · arXiv · doi:10.48550/arxiv.1803.03835 · Cited by 67 · see citing works
An Environment Model for Nonstationary Reinforcement Learning
Choi, S. P. M., Yeung, D., & Zhang, N. L. (1999) · Cited by 37 · see citing works
Investigating Generalisation in Continuous Deep Reinforcement LearningPreprint
Zhao, C., Sigaud, O., Stulp, F., & Hospedales, T. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07015 · Cited by 32 · see citing works
Regularization Matters in Policy OptimizationPreprint
Liu, Z., Li, X., Kang, B., & Darrell, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09191 · Cited by 25 · see citing works
Flagged first, then by citations
5 of the 36 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193 · see citing works
Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
Parker-Holder, J., Rajan, R., Song, X., Biedenkapp, A., Miao, Y., Eimer, T., Zhang, B., Nguyen, V., Calandra, R., Faust, A., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13596 · Cited by 98 · see citing works
A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning
Wang, Z., Yang, E., Shen, L., & Huang, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3498346 · Cited by 101 · see citing works
A Survey of Generalisation in Deep Reinforcement LearningPreprint
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2021) · arXiv (Cornell University) · Cited by 63 · see citing works
Mwata-Velu, T., Avina‐Cervantes, J. G., Cruz‐Duarte, J. M., Rostro‐González, H., & Ruiz-Pinales, J. (2021) · Mathematics · doi:10.3390/math9243297 · Cited by 19 · see citing works
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning AgentsPreprint
Powers, S., Xing, E., Kolve, E., Mottaghi, R., & Gupta, A. (2021) · CoLLAs · doi:10.48550/arxiv.2110.10067 · Cited by 52 · see citing works
A Comprehensive Study on Self-Learning Methods and Implications to Autonomous Driving
Xing, J., Wei, D., Zhou, S., Wang, T., Huang, Y., & Chen, H. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3440498 · Cited by 17 · see citing works
Ball, P. J., Lu, C., Parker-Holder, J., & Roberts, S. (2021) · arXiv · doi:10.48550/arxiv.2104.05632 · Cited by 11 · see citing works
SCRIPT: A Scalable Continual Reinforcement Learning Framework for Autonomous Penetration Testing
Zhou, S., Liu, J., Lu, Y., Yang, J., Zhang, Y., Lin, B., Zhong, X., & Hu, S. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127827 · Cited by 7 · see citing works
Reinforcement Learning by Guided Safe Exploration
Yang, Q., Simão, T. D., Jansen, N., Tindemans, S. H., & Spaan, M. T. J. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230598 · Cited by 6 · see citing works
Wang, Y., Yan, Z., Sang, L., Hong, L., Hu, Q., Shahidehpour, M., & Xu, Q. (2023) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2023.3262189 · Cited by 6 · see citing works
Hassan, M. A., & Lee, C.-G. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109057 · Cited by 5 · see citing works
Improving Sample Efficiency Through Stability Enhancement in Deep-Reinforcement Learning
Wang, Z., Jiang, W., Peng, R., Kou, Q., Wan, L., & Lan, X. (2025) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2025.3578050 · Cited by 4 · see citing works
Attention-based Partial Decoupling of Policy and Value for Generalization in Reinforcement Learning
Nafi, N. M., Glasscock, C., & Hsu, W. W. Y. (2022) · International Conference on Machine Learning and Applications · doi:10.1109/icmla55696.2022.00011 · Cited by 4 · see citing works
Jin, W., Tian, X., Wang, N., Wu, B., Shi, B., Zhao, B., & Yang, G. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107875 · Cited by 3 · see citing works
Xu, J.-Y., Hu, P., & Alrifaee, B. (2024) · 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc58415.2024.10919918 · Cited by 11 · see citing works
A study on the plasticity of neural networks
Berariu, T., Czarnecki, W., De, S., Bornschein, J., Smith, S., Pascanu, R., & Clopath, C. (2021) · arXiv · doi:10.48550/arxiv.2106.00042 · Cited by 3 · see citing works
Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks
Matthews, M., Beukman, M., Lu, C., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2410.23208 · Cited by 1 · see citing works
“…Kinetix is an excellent framework for investigating issues in agent training such as network capacity (Obando-Ceron et al., 2024), plasticity loss (Igl et al., 2020; Berariu et al., 2021; Sokar et al., 2023), lifelong learning (Kirkpatrick et al., 2017) and multi-task learning (Sodhani et al.,…”
- Authors
- Igl, M., Farquhar, G., Luketina, J., Boehmer, W., & Whiteson, S.
- Affiliations
- University of Oxford (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.05826
- OpenAlex
- W3118409705
- Cited by
- 17
- References
- 36
- Citations by year
- 2025: 3 · 2024: 4 · 2023: 3 · 2022: 2 · 2021: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Research Repository (Delft University of Technology) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Keywords
- deep reinforcement learning, generalization, knowledge transfer, memory effects, latent representations, behavioral public policy, Procgen
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
