The DOI matches its Crossref record, with a small difference in title or year; 435 of the 474 references it cites were found in a registry or an index.
Towards Continual Reinforcement Learning: A Review and Perspectives
Open accessAbstract
In this article, we aim to provide a literature review of different formulations and approaches to continual reinforcement learning (RL), also known as lifelong or non-stationary RL. We begin by discussing our perspective on why RL is a natural fit for studying continual learning. We then provide a taxonomy of different continual RL formulations by mathematically characterizing two key properties of non-stationarity, namely, the scope and driver non-stationarity. This offers a unified view of various formulations. Next, we review and present a taxonomy of continual RL approaches. We go on to discuss evaluation of continual RL agents, providing an overview of benchmarks used in the literature and important metrics for understanding agent performance. Finally, we highlight open problems and challenges in bridging the gap between the current state of continual RL and findings in neuroscience. While still in its early days, the study of continual RL has the promise to develop better incremental reinforcement learners that can function in increasingly realistic applications where non-stationarity plays a vital role. These include applications such as those in the fields of healthcare, education, logistics, and robotics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Another line develops affordance theory in RL, where the environment constrains and suggests possible actions (Graves et al., 2022; Khetarpal et al., 2020). Rather than treating all actions as always available, agents learn which are afforded in each situation, mirroring selective, skillful responses in enactive perception.”
From Toward Enactive Artificial Intelligence (2026) · cites this paper as Khetarpal et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While continual learning does not necessarily emphasize agent-gathered data or sustained interaction with the environment, it aligns with enactive views in emphasizing ongoing adaptation under non-stationarity.”
From Toward Enactive Artificial Intelligence (2026) · cites this paper as Khetarpal et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Research works printed but not linked: 26
- PMLR, 2021.
- David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continuum learning. Advances in Neural Information Processing Systems, 2017.
- Morgan Kaufmann Publishers Inc., 1994. ISBN 1-55860-335-2. URL http://dl.acm.org/citation.cfm?id=3091574.3091594.
- Qi Cai, Zhuoran Yang, and Zhaoran Wang. Sample-efficient reinforcement learning for pomdps with linear function approximations. arXiv preprint arXiv:2204.09787, 2022. · arXiv
- Weitong Zhang, Jiafan He, Dongruo Zhou, Amy Zhang, and Quanquan Gu. Provably efficient representation learning in low-rank markov decision processes. arXiv preprint arXiv:2106.11935, 2021a. · arXiv
- Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. arXiv preprint arXiv:2107.02729, 2021. · arXiv
- PMLR, 2022.
- Ayush Jain, Norio Kosaka, Kyung-Min Kim, and Joseph J Lim. Know your action set: Learning action relations for reinforcement learning. In International Conference on Learning Representations, 2021.
- Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
- Pascal Klink, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Boosted curriculum reinforce- ment learning. In International Conference on Learning Representations, 2021.
- Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836, 2020b.
- Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541. ACM, 2006.
- Maximilian Igl, Gregory Farquhar, Jelena Luketina, JW Böhmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In 9th International Conference on Learning Representations, 2021.
- Youngmin Oh, Jinwoo Shin, Eunho Yang, and Sung Ju Hwang. Model-augmented prioritized experience replay. In International Conference on Learning Representations, 2021.
- Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Reset-free lifelong learning with skill-space planning. In International Conference on Learning Representations, 2021.
- David Abel, Dilip Arumugam, Lucas Lehnert, and Michael L Littman. Toward good abstractions for lifelong learning. In NIPS Workshop on Hierarchical Reinforcement Learning, 2017.
- Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free explo- ration for reinforcement learning. In International Conference on Machine Learning, pages 4870-4879. PMLR, 2020.
- Dagui Chen, Qi Yan, Shangqi Guo, Zhile Yang, Xin Su, and Feng Chen. Learning effective subgoals with multi-task hierarchical reinforcement learning. Scaling-Up Reinforce- ment Learning (SURL) Workshop. URl: http://surl. tirl. info/proceedings/SURL- 2019 paper 10. pdf, 2019.
- Lorenzo Moro, Amarildo Likmeta, Enrico Prati, Marcello Restelli, et al. Goal-directed planning via hindsight experience replay. In International Conference on Learning Representations, pages 1-16, 2022.
- Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. arXiv preprint arXiv:1712.00948, 2017. · arXiv
- Curran Associates, Inc., 2019. URL http://papers.nips.cc/paper/8458-meta-learning-representations-for -continual-learning.pdf.
- Frederic Kaplan and Pierre-Yves Oudeyer. Curiosity-driven development. In Proceedings of the International Workshop on Synergistic Intelligence Dynamics. Citeseer, 2006.
- Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pages 1282-1289. PMLR, 2019a.
- Hiroshi Imamizu. Separated modules for visuomotor control and learning in the cerebellum: a functional mri study. In NeuroImage: Third International Conference on Functional Mapping of the Human Brain, volume 5, page S598. Academic Press, 1997.
- Morgan Kaufmann Publishers Inc. ISBN 1558607781.
- Amy Greenwald and Keith Hall. Correlated-Q learning. In ICML, page 242-249. AAAI Press, 2003. ISBN 1577351894.
Cited by
Sorted by citations
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113
Organoid intelligence (OI): the new frontier in biocomputing and intelligence-in-a-dish
Smirnova, L., Caffo, B. S., Gracias, D. H., Huang, Q., Pantoja, I. E. M., Tang, B., Zack, D. J., Berlinicke, C. A., Boyd, J. L., Harris, T. D., et al. (2023) · Frontiers in Science · doi:10.3389/fsci.2023.1017235 · Cited by 329
A Survey of Deep RL and IL for Autonomous Driving Policy Learning
Zhu, Z., & Zhao, H. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3134702 · Cited by 260
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193
“Khetarpal et al. (2020) motivated and surveyed continual reinforcement learning (CRL), which is closely related to ZSG in RL as both settings require adaptation to unseen tasks or environments; however, they explicitly do not discuss the zero-shot setting that is the concern of this paper (for more…”
BackgroundInfluentialAutonomous Drone Racing: A Survey
Hanover, D., Loquercio, A., Bauersfeld, L., Romero, Á., Pěnička, R., Song, Y., Cioffi, G., Kaufmann, E., & Scaramuzza, D. (2024) · IEEE Transactions on Robotics · doi:10.1109/tro.2024.3400838 · Cited by 173
Reinforcement Learning in Process Industries: Review and Perspective
Dogru, O., Xie, J., Prakash, O., Chiplunkar, R., Soesanto, J. F., Chen, H., Velswamy, K., Ibrahim, F., & Huang, B. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124227 · Cited by 124
Al-Hamadani, M. N. A., Fadhel, M. A., Alzubaidi, L., & Harangi, B. (2024) · Sensors · doi:10.3390/s24082461 · Cited by 109
A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning
Wang, Z., Yang, E., Shen, L., & Huang, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3498346 · Cited by 132
Continual learning for predictive maintenance: Overview and challenges
Hurtado, J., Salvati, D., Semola, R., Bosio, M., & Lomonaco, V. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2023.200251 · Cited by 70
Metaverse for Wireless Systems: Vision, Enablers, Architecture, and Future Directions
Khan, L. U., Han, Z., Niyato, D. T., Guizani, M., & Hong, C. S. (2024) · IEEE Wireless Communications · doi:10.1109/mwc.013.2300287 · Cited by 68
A Survey on Scheduling Techniques in Computing and Network Convergence
Tang, S., Yu, Y., Wang, H., Wang, G., Chen, W., Xu, Z., Guo, S., & Gao, W. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3329027 · Cited by 77
A Survey of Generalisation in Deep Reinforcement LearningPreprint
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2021) · arXiv (Cornell University) · Cited by 63
Continual Learning With Knowledge Distillation: A Survey
Li, S., Su, T., Zhang, X.-Y., & Wang, Z. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3476068 · Cited by 71
Yu, H., Shokrnezhad, M., Taleb, T., Li, R. Z., & Song, J. (2023) · IEEE Network · doi:10.1109/mnet.004.2300101 · Cited by 60
Continual Deep Learning for Time Series Modeling
Ao, S.-I., & Fayek, H. M. (2023) · Sensors · doi:10.3390/s23167167 · Cited by 64
Preserving and combining knowledge in robotic lifelong reinforcement learning
Meng, Y., Bing, Z., Yao, X., Chen, K., Huang, K., Gao, Y., Sun, F., & Knoll, A. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-00983-2 · Cited by 59
A collective AI via lifelong learning and sharing at the edge
Soltoggio, A., Ben-Iwhiwhu, E., Braverman, V., Eaton, E., Epstein, B., Ge, Y., Halperin, L., How, J., Itti, L., Jacobs, M. A., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00800-2 · Cited by 51
Attention-Based QoE-Aware Digital Twin Empowered Edge Computing for Immersive Virtual Reality
Yu, J., Alhilal, A. Y., Zhou, T., Hui, P., & Tsang, D. H. K. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3380820 · Cited by 50
Geng, B., Wu, D., Zhang, C., Xie, W., Mahmood, M. A., & Ali, Q. (2024) · Sustainability · doi:10.3390/su16010429 · Cited by 50
A rubric for human-like agents and NeuroAI
Momennejad, I. (2022) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2021.0446 · Cited by 42
Continual Learning for Smart City: A Survey
Yang, L., Luo, Z., Zhang, S., Teng, F., & Li, T. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3447123 · Cited by 46
A Definition of Continual Reinforcement Learning
Abel, D., Barreto, A., Roy, B. V., Precup, D., Hasselt, H. V., & Singh, S. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2192 · Cited by 36
Self-citationNeural circuits and symbolic processing
Do, Q. H., & Hasselmo, M. E. (2021) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2021.107552 · Cited by 38
Pareto: Fair Congestion Control With Online Reinforcement Learning
Emara, S., Fei, W., Li, B., & Zeyl, T. J. (2022) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2022.3185253 · Cited by 34
Measuring student attention based on EEG brain signals using deep reinforcement learning
Rehman, A. U., Shi, X., Ullah, F., Wang, Z., & Ma, C. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126426 · Cited by 36
A Survey of Machine Learning-Based Ride-Hailing Planning
Wen, D., Li, Y., & Lau, F. C. M. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3345174 · Cited by 33
Continual Deep Reinforcement Learning for Decentralized Satellite Routing
Lozano-Cuadra, F., Soret, B., Leyva‐Mayorga, I., & Popovski, P. (2025) · IEEE Transactions on Communications · doi:10.1109/tcomm.2025.3562522 · Cited by 36
Continual World: A Robotic Benchmark For Continual Reinforcement LearningPreprint
Wołczyk, M., Zając, M., Pascanu, R., Kuciński, Ł., & Miłoś, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.10919 · Cited by 25
Zhang, T., Wang, X., Liang, B., & Yuan, B. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3162241 · Cited by 24
Dvir, E., Shifrin, M., & Gurewitz, O. (2024) · Mathematics · doi:10.3390/math12132102 · Cited by 23
Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Review
Serrano, S. A., Martínez-Carranza, J., & Sucar, L. E. (2024) · IEEE Access · doi:10.1109/access.2024.3435558 · Cited by 26
Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignment
Ganjdanesh, A., Gao, S., & Huang, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01520 · Cited by 22
Huang, D., Zhao, H. P., Zhang, L., & Chen, K. (2024) · IEEE Access · doi:10.1109/access.2024.3384923 · Cited by 20
Continuum: Simple Management of Complex Continual Learning ScenariosPreprint
Douillard, A., & Lesort, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06253 · Cited by 30
Same State, Different Task: Continual Reinforcement Learning without Interference
Kessler, S., Parker-Holder, J., Ball, P., Zohren, S., & Roberts, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20674 · Cited by 23
Wei, W., Li, H., Zhou, S., Li, B., & Liu, X. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3563725 · Cited by 19
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning AgentsPreprint
Powers, S., Xing, E., Kolve, E., Mottaghi, R., & Gupta, A. (2021) · CoLLAs · doi:10.48550/arxiv.2110.10067 · Cited by 52
Paesschesoone, S., Kayedpour, N., Manna, C., & Crevecoeur, G. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.123507 · Cited by 17
ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format
Zhu, Q., Geishauser, C., Lin, H.-C., van Niekerk, C., Peng, B., Zhang, Z., Feng, S., Heck, M. C., Lubis, N., Wan, D., et al. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.9 · Cited by 16
Learning to Resolve Social Dilemmas: A Survey
Fatima, S. S., Jennings, N. R., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15167 · Cited by 16
Universal Off-Policy EvaluationPreprint
Chandak, Y., Niekum, S., da Silva, B. C., Miller, E. G., Brunskill, E., & Thomas, P. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.12820 · Cited by 16
Continual learning: A systematic literature review
Yang, Q., Wang, L., Wicker, J., & Dobbie, G. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108226 · Cited by 23
Phyllis: Physics-Informed Lifelong Reinforcement Learning for Data Center Cooling Control
Wang, R., Cao, Z., Zhou, X., Wen, Y., & Tan, R. (2023) · ACM International Conference on Future Energy Systems (e-Energy) · doi:10.1145/3575813.3595189 · Cited by 15
Wasse, A. D., Dai, K., Wang, J., & Sharbati, R. (2024) · International Journal of Steel Structures · doi:10.1007/s13296-024-00815-w · Cited by 15
A sparse quantized hopfield network for online-continual memory
Alonso, N., & Krichmar, J. L. (2024) · Nature Communications · doi:10.1038/s41467-024-46976-4 · Cited by 17
Sampled-data funnel control and its use for safe continual learning
Lanza, L., Dennstädt, D., Worthmann, K., Schmitz, P., Şen, G. D., Trenn, S., & Schaller, M. (2024) · Systems & Control Letters · doi:10.1016/j.sysconle.2024.105892 · Cited by 15
Abernot, M., Azémard, N., & Todri‐Sanial, A. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1196796 · Cited by 19
Hamilton-Jacobi Reachability in Reinforcement Learning: A Survey
Ganai, M., Gao, S., & Herbert, S. (2024) · IEEE Open Journal of Control Systems · doi:10.1109/ojcsys.2024.3449138 · Cited by 20
Continuous Coordination As a Realistic Scenario for Lifelong LearningPreprint
Nekoei, H., Badrinaaraayanan, A., Courville, A. C., & Chandar, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03216 · Cited by 17
Continual Backprop: Stochastic Gradient Descent with Persistent RandomnessPreprint
Dohare, S., Sutton, R. S., & Mahmood, A. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.06325 · Cited by 88
Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization
Zhang, T., Lin, Z., Wang, Y., Ye, D., Fu, Q., Yang, W., Wang, X., Liang, B., Yuan, B., & Li, X. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3280085 · Cited by 13
Deep Reinforcement Learning in Nonstationary Environments With Unknown Change Points
Liu, Z. Q., Lü, J., Xuan, J., & Zhang, G. (2024) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2024.3356981 · Cited by 14
Reinforcement learning in cold atom experiments
Reinschmidt, M., Fortágh, J., Günther, A., & Volchkov, V. (2024) · Nature Communications · doi:10.1038/s41467-024-52775-8 · Cited by 24
Afolayan, B. I., Ghosh, A., Calderín, J. F., & Masegosa, A. D. (2024) · IEEE Access · doi:10.1109/access.2024.3501775 · Cited by 13
Prevalence of Code Smells in Reinforcement Learning Projects
Cardozo, N., Dusparić, I., & Cabrera, C. (2023) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain58948.2023.00013 · Cited by 12
Mazandarani, H., Shokrnezhad, M., Taleb, T., & Li, R. Z. (2023) · IEEE International Conference on Metaverse Computing, Networking and Applications (MetaCom) · doi:10.1109/metacom57706.2023.00024 · Cited by 12
Visual adversarial imitation learning using variational models
Rafailov, R., Yu, T., Rajeswaran, A., & Finn, C. (2021) · arXiv · doi:10.48550/arxiv.2107.08829 · Cited by 12
Campbell, E., Egle, F., Oßwald, M., Côté‐Allard, U., Pilarski, P. M., Boccardo, N., Meattini, R., Vujaklija, I., Hargrove, L. J., Canepa, M., et al. (2025) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2025.3602397 · Cited by 14
Li, J., Xie, H., Zhang, J., Li, L., Song, G., Fu, H., Chen, P., Liu, C., Zhang, L., Shi, Z., et al. (2025) · Advances in Applied Energy · doi:10.1016/j.adapen.2025.100239 · Cited by 12
Özsoydan, F. B. (2025) · Computers & Industrial Engineering · doi:10.1016/j.cie.2025.110948 · Cited by 11
Naser, M. Z. (2025) · Machine learning for computational science and engineering · doi:10.1007/s44379-025-00020-4 · Cited by 11
Diabetes-focused food recommender system (DFRS) to enabling digital health
Ahmed, E., Oumer, M., & Hassan, M. S. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000530 · Cited by 10
Hu, J., Hendrix, R. C., Farhadi, A., Kembhavi, A., Martín-Martín, R., Stone, P., Zeng, K.-H., & Ehsani, K. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11127934 · Cited by 40
Sequoia: A Software Framework to Unify Continual Learning ResearchPreprint
Normandin, F., Golemo, F., Ostapenko, O., Rodríguez, P., Riemer, M., Hurtado, J., Khetarpal, K., Lindeborg, R., Cecchi, L., Lesort, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01005 · Cited by 12
Self-citationACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
Wu, K., Zhao, Y., Xu, Z., Che, Z., Yin, C., Liu, C. H., Feng, F., & Tang, J. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3497667 · Cited by 13
Autonomous Reinforcement Learning via Subgoal CurriculaPreprint
Sharma, A., Gupta, A., Levine, S., Hausman, K., & Finn, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.12931 · Cited by 9
Kim, B.-S., Suk, H.-W., Choi, Y., Moon, D., & Kim, M. S. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2024.052375 · Cited by 11
Wong, P. Y. L., Lai, J. H. K., & Lo, K. C. C. (2025) · Sustainability · doi:10.3390/su17030881 · Cited by 8
Loss of Plasticity in Continual Deep Reinforcement Learning
Abbas, Z., Zhao, R., Modayil, J., White, A., & Machado, M. C. (2023) · arXiv · doi:10.48550/arxiv.2303.07507 · Cited by 10
Reservoir computing with brain organoids
Smirnova, L., Caffo, B. S., & Johnson, E. C. (2023) · Nature Electronics · doi:10.1038/s41928-023-01096-7 · Cited by 7
Continual Domain Randomization
Josifovski, J., Auddy, S., Malmir, M., Piater, J., Knoll, A., & Navarro-Guerrero, N. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802060 · Cited by 7
SCRIPT: A Scalable Continual Reinforcement Learning Framework for Autonomous Penetration Testing
Zhou, S., Liu, J., Lu, Y., Yang, J., Zhang, Y., Lin, B., Zhong, X., & Hu, S. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127827 · Cited by 10
Ada, S. E., Say, H., Uğur, E., & Öztop, E. (2024) · IEEE Access · doi:10.1109/access.2024.3402089 · Cited by 7
A Survey: Navigating the Landscape of Incremental Learning Techniques and Trends
Li, Y., Wu, W., Luo, X., Zheng, M., Zhang, Y., & Peng, B. (2023) · International Conference on Intelligent Systems and Knowledge Engineering (ISKE) · doi:10.1109/iske60036.2023.10481497 · Cited by 7
Kumar, P., Hota, L., Nayak, B. P., & Sangaiah, A. K. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.04.288 · Cited by 7
A Workflow for Building Computationally Rational Models of Human Behavior
Chandramouli, S., Shi, D., Putkonen, A., Peuter, S. D., Zhang, S., Jokinen, J., Howes, A., & Oulasvirta, A. (2024) · Computational Brain & Behavior · doi:10.1007/s42113-024-00208-6 · Cited by 11
Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization
Liotet, P., Vidaich, F., Metelli, A. M., & Restelli, M. (2022) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v36i7.20717 · Cited by 7
Avalanche: an End-to-End Library for Continual Learning
Lomonaco, V., Pellegrini, L., Cossu, A., Carta, A., Graffieti, G., Hayes, T. L., Lange, M. D., Masana, M., Pomponi, J., van de Ven, G. M., et al. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00399 · Cited by 137
Mazandarani, H., Shokrnezhad, M., & Taleb, T. (2025) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2025.3546183 · Cited by 8
Chen, J., Zhao, C., Gao, K., Ji, Y., & Du, Y. (2025) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.13503 · Cited by 6
Beyond Reinforcement Learning for network security: A comprehensive survey and tutorial
Javadpour, A., Ja’fari, F., Taleb, T., Türkmen, F., & Benzaïd, C. (2025) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2025.104294 · Cited by 6
CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
Liu, Y., Guo, B., Li, N., Ding, Y., Zhang, Z., & Yu, Z. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3423319 · Cited by 13
Cardozo, N., & Dusparić, I. (2021) · International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) · doi:10.1109/seams51251.2021.00041 · Cited by 6
Surfing Information: The Challenge of Intelligent Decision-Making
Wu, C., & Zhang, Z. (2023) · Intelligent Computing · doi:10.34133/icomputing.0041 · Cited by 6
Liu, Y., Jiang, X., Yin, Y., Wang, Y., Mi, F., Liu, Q., Wan, X., & Wang, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.1 · Cited by 6
Emergent Communication in Multi-Agent Reinforcement Learning for Flying Base Stations
Naoumi, S., Alami, R., Hacid, H., Almazrouei, E., Debbah, M. A., Bennis, M., & Chafii, M. (2023) · IEEE International Mediterranean Conference on Communications and Networking (MeditCom) · doi:10.1109/meditcom58224.2023.10266608 · Cited by 6
Vantilborgh, V., Staessens, T., Groote, W. D., & Crevecoeur, G. (2023) · Control Engineering Practice · doi:10.1016/j.conengprac.2023.105783 · Cited by 6
Predictive reinforcement learning in non-stationary environments using weighted mixture policy
Pourshamsaei, H., & Nobakhti, A. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111305 · Cited by 7
A Study of Continual Learning Methods for Q-Learning
Bagus, B., & Gepperth, A. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892384 · Cited by 6
Vibration Suppression of Robotic Manipulators: A Review
Xu, D., Yang, G., Feng, H., Wang, X., & Liu, Y. (2025) · IEEE Open Journal of the Industrial Electronics Society · doi:10.1109/ojies.2025.3610876 · Cited by 5
Advanced EEG Signal Processing and Deep Q-Learning for Accurate Student Attention Monitoring
Rehman, A. U., Eman, N. U., Ullah, F., Cacciagrano, D., Mostarda, L., & Raza, N. M. (2024) · IEEE Access · doi:10.1109/access.2024.3523225 · Cited by 5
GAPPO-ERW: an imitation reinforcement learning approach for AGV path planning
Chen, W., Lin, X., Zhou, Z., Qiao, Y., Li, P., & Yu, D. X. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07361-0 · Cited by 5
Multiagent Continual Coordination via Progressive Task Contextualization
Yuan, L., Li, L., Zhang, Z., Zhang, F., Guan, C., & Yu, Y. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3394513 · Cited by 8
Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policies
Lawson, D., & Qureshi, A. H. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610919 · Cited by 5
High-Efficiency Vector Field by Time-Optimal Spatial Iterative Learning
Lv, S., Gao, Y., & Quan, Q. (2025) · IEEE Transactions on Robotics · doi:10.1109/tro.2025.3610174 · Cited by 4
Learning With an Open Horizon in Ever-Changing Dialogue Circumstances
Geishauser, C., van Niekerk, C., Lubis, N., Lin, H.-C., Heck, M. C., Feng, S., Ruppik, B., Vukovic, R., & Gašić, M. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3385289 · Cited by 5
Survival games for humans and machines
Strannegård, C., Engsner, N., Ulfsbäcker, S., Andreasson, S., Endler, J. A., & Nordgren, A. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2024.101235 · Cited by 4
Hybrid Control Framework of UAVs Under Varying Wind and Payload Conditions
Coursey, A., Zhang, A., Quiñones-Grueiro, M., & Biswas, G. (2024) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc60939.2024.10645000 · Cited by 4
Cross-Domain DRL Agents for Efficient Job Placement in the Cloud-Edge Continuum
Aslanidis, T., Kosta, S., Lalis, S., & Chatzopoulos, D. (2025) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3721146.3721934 · Cited by 5
Continual Learning as Computationally Constrained Reinforcement Learning
Kumar, S., Marklund, H., Rao, A., Zhu, Y., Jeon, H. J., Liu, Y., & Roy, B. V. (2025) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000116 · Cited by 36
Metadata
- Authors
- Khetarpal, K., Riemer, M., Rish, I., & Precup, D.
- Affiliations
- McGill University (CA)
- Mila - Quebec Artificial Intelligence Institute (CA)
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2022-12-22
- Volume
- 75
- Pages
- 1401-1476
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W3117215073
- Cited by
- 224
- References
- 474
- Citations by year
- 2026: 37 · 2025: 54 · 2024: 72 · 2023: 28 · 2022: 11
- FWCI
- 13.57
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Artificial Intelligence Research (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Canadian Institute for Advanced Research
- Canada Excellence Research Chairs, Government of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Keywords
- continual reinforcement learning, lifelong learning, incremental learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
