The DOI matches its Crossref record, with a small difference in title or year; 435 of the 474 references it cites were found in a registry or an index.
Towards Continual Reinforcement Learning: A Review and Perspectives
Open accessAbstract
In this article, we aim to provide a literature review of different formulations and approaches to continual reinforcement learning (RL), also known as lifelong or non-stationary RL. We begin by discussing our perspective on why RL is a natural fit for studying continual learning. We then provide a taxonomy of different continual RL formulations by mathematically characterizing two key properties of non-stationarity, namely, the scope and driver non-stationarity. This offers a unified view of various formulations. Next, we review and present a taxonomy of continual RL approaches. We go on to discuss evaluation of continual RL agents, providing an overview of benchmarks used in the literature and important metrics for understanding agent performance. Finally, we highlight open problems and challenges in bridging the gap between the current state of continual RL and findings in neuroscience. While still in its early days, the study of continual RL has the promise to develop better incremental reinforcement learners that can function in increasingly realistic applications where non-stationarity plays a vital role. These include applications such as those in the fields of healthcare, education, logistics, and robotics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Another line develops affordance theory in RL, where the environment constrains and suggests possible actions (Graves et al., 2022; Khetarpal et al., 2020). Rather than treating all actions as always available, agents learn which are afforded in each situation, mirroring selective, skillful responses in enactive perception.”
From Toward Enactive Artificial Intelligence (2026) · cites this paper as Khetarpal et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While continual learning does not necessarily emphasize agent-gathered data or sustained interaction with the environment, it aligns with enactive views in emphasizing ongoing adaptation under non-stationarity.”
From Toward Enactive Artificial Intelligence (2026) · cites this paper as Khetarpal et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 401–435 of 435
Representation Learning for Online and Offline RL in Low-rank MDPsPreprint
Uehara, M., Zhang, X., & Sun, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04652 · Cited by 7
Reward-Free Model-Based Reinforcement Learning with Linear Function ApproximationPreprint
Zhang, W., Zhou, D., & Gu, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06394 · Cited by 7
Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization
Liotet, P., Vidaich, F., Metelli, A. M., & Restelli, M. (2022) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v36i7.20717 · Cited by 7
Influencing Long-Term Behavior in Multiagent Reinforcement Learning
Kim, D.-K., Riemer, M., Liu, M., Foerster, J., Everett, M., Sun, C., Tesauro, G. J., & How, J. P. (2022) · neural information processing systems · doi:10.52202/068431-1366 · Cited by 7
Self-citationAnyMorph: Learning Transferable Polices By Inferring Agent MorphologyPreprint
Trabucco, B., Phielipp, M., & Berseth, G. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.12279 · Cited by 7
Context-Specific Representation Abstraction for Deep Option Learning
Abdulhai, M., Kim, D.-K., Riemer, M., Liu, M., Tesauro, G. J., & How, J. P. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i6.20541 · Cited by 6
Self-citationLearning One Representation to Optimize All RewardsPreprint
Touati, A., & Ollivier, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.07945 · Cited by 6
Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2Preprint
Daniels, Z., Raghavan, A., Hostetler, J., Rahman, A., Sur, I., Piacentino, M., & Divakaran, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.05056 · Cited by 6
Policy Gradients Incorporating the FuturePreprint
Venuto, D., Lau, E., Precup, D., & Nachum, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02096 · Cited by 5
Self-citationLearning Structures: Predictive Representations, Replay, and GeneralizationPreprint
Momennejad, I. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/b6sr8 · Cited by 5
On Value Functions and the Agent-Environment Boundary
Jiang, N. (2019) · arXiv · doi:10.48550/arxiv.1905.13341 · Cited by 5
How RL Agents Behave When Their Actions Are Modified
Langlois, E. D., & Everitt, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i13.17378 · Cited by 5
Monte Carlo Tree Search With Iteratively Refining State Abstractions
Sokota, S., Ho, C., Ahmad, Z. F., & Kolter, J. Z. (2021) · neural information processing systems · Cited by 5
Learning Domain Invariant Representations in Goal-conditioned Block MDPsPreprint
Han, B., Zheng, C., Chan, H., Paster, K., Zhang, M. R., & Ba, J. L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.14248 · Cited by 5
The Role of Executive Function in Shaping Reinforcement LearningPreprint
Rmus, M., McDougle, S. D., & Collins, A. G. E. (2020) · doi:10.31234/osf.io/9cvw3 · Cited by 5
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics MixturePreprint
Lee, S., & Chung, S.-Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13524 · Cited by 4
The Barbados 2018 List of Open Issues in Continual LearningPreprint
Schaul, T., van Hasselt, H. P., Modayil, J., White, M., White, A., Bacon, P., Harb, J., Mourad, S., Bellemare, M. G., & Precup, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.07004 · Cited by 4
Self-citationGoal-Aware Cross-Entropy for Multi-Target Reinforcement LearningPreprint
Kim, K., Lee, M. W., Kim, Y., Ryu, J.-H., Lee, M. S., & Zhang, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12985 · Cited by 4
Sparsely Changing Latent States for Prediction and Planning in Partially Observable DomainsPreprint
Gumbsch, C., Butz, M. V., & Martius, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15949 · Cited by 4
Memory-efficient Reinforcement Learning with Value-based Knowledge ConsolidationPreprint
Lan, Q., Pan, Y., Luo, J., & Mahmood, A. R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.10868 · Cited by 4
Hindsight Foresight Relabeling for Meta-Reinforcement LearningPreprint
Wan, M., Peng, J., & Gangwani, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.09031 · Cited by 4
Accelerating Online Reinforcement Learning via Model-Based Meta-Learning
Co-Reyes, J. D., Feng, S., Berseth, G., Qui, J. F., & Levine, S. (2021) · International Conference on Learning Representations · Cited by 3
Provable Rich Observation Reinforcement Learning with Combinatorial Latent States
Misra, D. K., Liu, Q., Jin, C., & Langford, J. C. (2021) · International Conference on Learning Representations · Cited by 3
Block Contextual MDPs for Continual Learning
Sodhani, S., Meier, F., Pineau, J., & Zhang, A. (2021) · arXiv · doi:10.48550/arxiv.2110.06972 · Cited by 3
L2Explorer: A Lifelong Reinforcement Learning Assessment EnvironmentPreprint
Johnson, E. C., Nguyen, E. Q., Schreurs, B. A., Ewulum, C., Ashcraft, C., Fendley, N., Baker, M. M., New, A., & Vallabha, G. K. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.07454 · Cited by 3
Toward Robust Long Range Policy Transfer
Tseng, W.-C., Lin, J.-S., Feng, Y.-M., & Sun, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i11.17196 · Cited by 2
Continual Reinforcement Learning with Diversity Exploration and Adversarial Self-CorrectionPreprint
Zhu, F., Chang, X., Zeng, R., & Tan, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09205 · Cited by 2
Online Learned Continual Compression with Stacked Quantization Module.Preprint
Caccia, L., Belilovsky, E., Caccia, M., & Pineau, J. (2019) · arXiv (Cornell University) · Cited by 2
Continual Learning In Environments With Polynomial Mixing TimesPreprint
Riemer, M., Raparthy, S. C., Cases, I., Subbaraj, G., Touzel, M. P., & Rish, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.07066 · Cited by 2
Self-citationInteractive Agent Modeling by Learning to ProbePreprint
Shu, T., Xiong, C., Wu, Y. N., & Zhu, S. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.00510 · Cited by 1
On the Role of Weight Sharing During Deep Option Learning
Riemer, M., Cases, I., Rosenbaum, C., Liu, M., & Tesauro, G. J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6003 · Cited by 1
Self-citationAutomated curriculum generation for Policy Gradients from DemonstrationsPreprint
Srinivasan, A., Bahdanau, D., Chevalier-Boisvert, M., & Bengio, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.00444 · Cited by 1
Structural Similarity for Improved Transfer in Reinforcement LearningPreprint
Ashcraft, C., Stoler, B., Ewulum, C., & Agarwala, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.13813 · Cited by 1
Quantifying Generalization in Reinforcement Learning
Cobbe, K., Klimov, O., Hesse, C., Kim, T., & Schulman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.02341
Flagged first, then by citations
39 of the 474 references couldn’t be matched to a known paper.
Research works printed but not linked: 26
- PMLR, 2021.
- David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continuum learning. Advances in Neural Information Processing Systems, 2017.
- Morgan Kaufmann Publishers Inc., 1994. ISBN 1-55860-335-2. URL http://dl.acm.org/citation.cfm?id=3091574.3091594.
- Qi Cai, Zhuoran Yang, and Zhaoran Wang. Sample-efficient reinforcement learning for pomdps with linear function approximations. arXiv preprint arXiv:2204.09787, 2022. · arXiv
- Weitong Zhang, Jiafan He, Dongruo Zhou, Amy Zhang, and Quanquan Gu. Provably efficient representation learning in low-rank markov decision processes. arXiv preprint arXiv:2106.11935, 2021a. · arXiv
- Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. arXiv preprint arXiv:2107.02729, 2021. · arXiv
- PMLR, 2022.
- Ayush Jain, Norio Kosaka, Kyung-Min Kim, and Joseph J Lim. Know your action set: Learning action relations for reinforcement learning. In International Conference on Learning Representations, 2021.
- Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
- Pascal Klink, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Boosted curriculum reinforce- ment learning. In International Conference on Learning Representations, 2021.
- Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836, 2020b.
- Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541. ACM, 2006.
- Maximilian Igl, Gregory Farquhar, Jelena Luketina, JW Böhmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In 9th International Conference on Learning Representations, 2021.
- Youngmin Oh, Jinwoo Shin, Eunho Yang, and Sung Ju Hwang. Model-augmented prioritized experience replay. In International Conference on Learning Representations, 2021.
- Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Reset-free lifelong learning with skill-space planning. In International Conference on Learning Representations, 2021.
- David Abel, Dilip Arumugam, Lucas Lehnert, and Michael L Littman. Toward good abstractions for lifelong learning. In NIPS Workshop on Hierarchical Reinforcement Learning, 2017.
- Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free explo- ration for reinforcement learning. In International Conference on Machine Learning, pages 4870-4879. PMLR, 2020.
- Dagui Chen, Qi Yan, Shangqi Guo, Zhile Yang, Xin Su, and Feng Chen. Learning effective subgoals with multi-task hierarchical reinforcement learning. Scaling-Up Reinforce- ment Learning (SURL) Workshop. URl: http://surl. tirl. info/proceedings/SURL- 2019 paper 10. pdf, 2019.
- Lorenzo Moro, Amarildo Likmeta, Enrico Prati, Marcello Restelli, et al. Goal-directed planning via hindsight experience replay. In International Conference on Learning Representations, pages 1-16, 2022.
- Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. arXiv preprint arXiv:1712.00948, 2017. · arXiv
- Curran Associates, Inc., 2019. URL http://papers.nips.cc/paper/8458-meta-learning-representations-for -continual-learning.pdf.
- Frederic Kaplan and Pierre-Yves Oudeyer. Curiosity-driven development. In Proceedings of the International Workshop on Synergistic Intelligence Dynamics. Citeseer, 2006.
- Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pages 1282-1289. PMLR, 2019a.
- Hiroshi Imamizu. Separated modules for visuomotor control and learning in the cerebellum: a functional mri study. In NeuroImage: Third International Conference on Functional Mapping of the Human Brain, volume 5, page S598. Academic Press, 1997.
- Morgan Kaufmann Publishers Inc. ISBN 1558607781.
- Amy Greenwald and Keith Hall. Correlated-Q learning. In ICML, page 242-249. AAAI Press, 2003. ISBN 1577351894.
Cited by
Sorted by citations
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113
Organoid intelligence (OI): the new frontier in biocomputing and intelligence-in-a-dish
Smirnova, L., Caffo, B. S., Gracias, D. H., Huang, Q., Pantoja, I. E. M., Tang, B., Zack, D. J., Berlinicke, C. A., Boyd, J. L., Harris, T. D., et al. (2023) · Frontiers in Science · doi:10.3389/fsci.2023.1017235 · Cited by 329
A Survey of Deep RL and IL for Autonomous Driving Policy Learning
Zhu, Z., & Zhao, H. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3134702 · Cited by 260
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193
“Khetarpal et al. (2020) motivated and surveyed continual reinforcement learning (CRL), which is closely related to ZSG in RL as both settings require adaptation to unseen tasks or environments; however, they explicitly do not discuss the zero-shot setting that is the concern of this paper (for more…”
BackgroundInfluentialAutonomous Drone Racing: A Survey
Hanover, D., Loquercio, A., Bauersfeld, L., Romero, Á., Pěnička, R., Song, Y., Cioffi, G., Kaufmann, E., & Scaramuzza, D. (2024) · IEEE Transactions on Robotics · doi:10.1109/tro.2024.3400838 · Cited by 173
Metadata
- Authors
- Khetarpal, K., Riemer, M., Rish, I., & Precup, D.
- Affiliations
- McGill University (CA)
- Mila - Quebec Artificial Intelligence Institute (CA)
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2022-12-22
- Volume
- 75
- Pages
- 1401-1476
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W3117215073
- Cited by
- 224
- References
- 474
- Citations by year
- 2026: 37 · 2025: 54 · 2024: 72 · 2023: 28 · 2022: 11
- FWCI
- 13.57
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Artificial Intelligence Research (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Canadian Institute for Advanced Research
- Canada Excellence Research Chairs, Government of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Keywords
- continual reinforcement learning, lifelong learning, incremental learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
