CoChat
Sign Up
PreprintCited by 227Reinforcement Learning in RoboticsDomain Adaptation and Few-Shot LearningAdvanced Bandit Algorithms Research

Towards Continual Reinforcement Learning: A Review and Perspectives

Open access
Khetarpal, Khimya, Riemer, Matthew, Rish, Irina, Precup, Doina·2020·arXiv
DOI 10.48550/arxiv.2012.13490arXiv 2012.13490
Open this paper in CoChat — free accountOpen PDF

Abstract

In this article, we aim to provide a literature review of different formulations and approaches to continual reinforcement learning (RL), also known as lifelong or non-stationary RL. We begin by discussing our perspective on why RL is a natural fit for studying continual learning. We then provide a taxonomy of different continual RL formulations by mathematically characterizing two key properties of non-stationarity, namely, the scope and driver non-stationarity. This offers a unified view of various formulations. Next, we review and present a taxonomy of continual RL approaches. We go on to discuss evaluation of continual RL agents, providing an overview of benchmarks used in the literature and important metrics for understanding agent performance. Finally, we highlight open problems and challenges in bridging the gap between the current state of continual RL and findings in neuroscience. While still in its early days, the study of continual RL has the promise to develop better incremental reinforcement learners that can function in increasingly realistic applications where non-stationarity plays a vital role. These include applications such as those in the fields of healthcare, education, logistics, and robotics.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2012.13490

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • None of the previous surveys of related RL subfields have covered zero-shot generalisation in RL explicitly.

    “While there have been previous surveys of related subfields in RL, none have covered zero-shot generalisation in RL explicitly.”

    From A Survey of Zero-shot Generalisation in Deep Reinforcement Learning (2023) · cites this paper as Khetarpal et al. (2020)

    The Key Contributions section identifies this paper as a survey of continual RL and defines its scope around topics other than zero-shot generalisation. Section 6.3 briefly mentions zero-shot expected returns as an evaluation metric, but the paper does not establish the broader claim that no previous survey explicitly covered zero-shot generalisation in RL. The indexed passages provide section labels but no page numbers. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • There are not enough good benchmarks for CRL.

    “we agree with Khetarpal et al. (2020) that there are not enough good benchmarks for CRL.”

    From A Survey of Zero-shot Generalisation in Deep Reinforcement Learning (2023) · cites this paper as Khetarpal et al. (2020)

  • Continual reinforcement learning (CRL) is closely related to ZSG in RL as both settings require adaptation to unseen tasks or environments.

    “Khetarpal et al. (2020) motivated and surveyed continual reinforcement learning (CRL), which is closely related to ZSG in RL as both settings require adaptation to unseen tasks or environments”

    From A Survey of Zero-shot Generalisation in Deep Reinforcement Learning (2023) · cites this paper as Khetarpal et al. (2020)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Generalization and Regularization in DQN

    Farebrother, J., Machado, M. C., & Bowling, M. (2018) · arXiv · doi:10.48550/arxiv.1810.00123 · Cited by 92 · cited 2× · see citing works

    “Generalization in RL has been investigated by creating different game modes (Farebrother, Machado, and Bowling, 2018) and game levels (Nichol, Pfau, Hesse, Klimov, and Schulman, 2018) for training and testing, by exposing internal parameters of various classic environments (Packer, Gao, Kos,…”
    Background

Flagged first, then by citations

Research works printed but not linked: 426

  1. Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. arXiv preprint arXiv:2003.06350, 2020. · arXiv
  2. Mark B Ring. Child: A first step towards continual learning. Machine Learning, 28(1): 77-104, 1997.
  3. German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
  4. Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting in classification tasks. arXiv preprint arXiv:1909.08383, 2019. · arXiv
  5. Martin Mundt, Yong Won Hong, Iuliia Pliushch, and Visvanathan Ramesh. A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning. arXiv preprint arXiv:2009.01797, 2020. · arXiv
  6. Raia Hadsell, Dushyant Rao, Andrei A. Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 24(12): 1028 -1040, 2020. ISSN 1364-6613. https://doi.org/10.1016/j.tics.2020.09.004. URL http://www.sciencedirect.com/science/article/pii/S1364661320302199. · DOI · Paper page
  7. ML Puterman. Markov decision processes. 1994. Jhon Wiley & Sons, New Jersey, 1994.
  8. Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
  9. Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, pages 1094-1099.
  10. Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst continual structured non-stationarity. In International Conference on Machine Learning, pages 11393-11403.
  11. PMLR, 2021.
  12. Matthew Riemer, Sharath Chandra Raparthy, Ignacio Cases, Gopeshh Subbaraj, Maximil- ian Puelma Touzel, and Irina Rish. Continual learning in environments with polynomial mixing times. Advances in Neural Information Processing Systems, 2022.
  13. Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(Jul):1633-1685, 2009.
  14. Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning, pages 1015-1022. ACM, 2007.
  15. Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew Taylor. Online multi-task learning for policy gradient methods. In International Conference on Machine Learning, pages 1206-1214, 2014.
  16. Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. · arXiv
  17. Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprint arXiv:1804.03720, 2018. · arXiv
  18. Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv:1810.12282, 2018. · arXiv
  19. Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint arXiv:1806.10729, 2018. · arXiv
  20. Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.06893, 2018a. · arXiv
  21. Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. arXiv preprint arXiv:1812.02341, 2018. · arXiv
  22. Simon S Du, Sham M Kakade, Ruosong Wang, and Lin F Yang. Is a good representation sufficient for sample efficient reinforcement learning? arXiv preprint arXiv:1910.03016, 2019a. · arXiv
  23. Gail A Carpenter and Stephen Grossberg. A massively parallel architecture for a self- organizing neural pattern recognition machine. Computer vision, graphics, and image processing, 37(1):54-115, 1987.
  24. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526, 2017.
  25. David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continuum learning. Advances in Neural Information Processing Systems, 2017.
  26. Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. ICLR, 2019.
  27. Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. ICLR, 2019.
  28. Weichao Mao, Kaiqing Zhang, Ruihao Zhu, David Simchi-Levi, and Tamer Basar. Near- optimal model-free reinforcement learning in non-stationary episodic mdps. In Inter- national Conference on Machine Learning, pages 7447-7458. PMLR, 2021.
  29. Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In ICML, pages 157-163.
  30. Morgan Kaufmann Publishers Inc., 1994. ISBN 1-55860-335-2. URL http://dl.acm.org/citation.cfm?id=3091574.3091594.
  31. Dong Ki Kim, Matthew Riemer, Miao Liu, Jakob N Foerster, Michael Everett, Chuangchuang Sun, Gerald Tesauro, and Jonathan P How. Influencing long-term behavior in mul- tiagent reinforcement learning. Advances in Neural Information Processing Systems, 2022.
  32. Finale Doshi-Velez and George Konidaris. Hidden parameter markov decision processes: A semiparametric regression approach for discovering latent task parametrizations. arXiv preprint arXiv:1308.3513, 2013. · arXiv
  33. Samuel PM Choi, Dit-Yan Yeung, and Nevin L Zhang. Hidden-mode markov decision processes for nonstationary sequential decision making. In Sequence Learning, pages 264-287. Springer, 2000.
  34. Sylvie CW Ong, Shao Wei Png, David Hsu, and Wee Sun Lee. Planning under uncertainty for robotic tasks with mixed observability. The International Journal of Robotics Research, 29(8):1053-1068, 2010.
  35. Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforce- ment learning. Journal of Machine Learning Research, 11(4), 2010.
  36. Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263-272. PMLR, 2017.
  37. Simon S Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudík, and John Langford. Provably efficient rl with rich observations via latent state decoding. arXiv preprint arXiv:1901.09018, 2019b. · arXiv
  38. Christian Gumbsch, Martin V Butz, and Georg Martius. Sparsely changing latent states for prediction and planning in partially observable domains. Advances in Neural Information Processing Systems, 34:17518-17531, 2021.
  39. Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored mdps. In IJCAI, volume 16, pages 740-747, 1999.
  40. Craig Boutilier, Richard Dearden, and Moisés Goldszmidt. Stochastic dynamic program- ming with factored representations. Artificial Intelligence, 121(1):49-107, 2000. ISSN 0004-3702. https://doi.org/10.1016/S0004-3702(00)00033-3. URL https:// www.sciencedirect.com/science/article/pii/S0004370200000333. · DOI · Paper page
  41. Qi Cai, Zhuoran Yang, and Zhaoran Wang. Sample-efficient reinforcement learning for pomdps with linear function approximations. arXiv preprint arXiv:2204.09787, 2022. · arXiv
  42. Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Struc- tural complexity and representation learning of low rank mdps. Advances in neural information processing systems, 33:20095-20107, 2020.
  43. Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline rl in low-rank mdps. arXiv preprint arXiv:2110.04652, 2021. · arXiv
  44. Weitong Zhang, Jiafan He, Dongruo Zhou, Amy Zhang, and Quanquan Gu. Provably efficient representation learning in low-rank markov decision processes. arXiv preprint arXiv:2106.11935, 2021a. · arXiv
  45. Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems, 34:13406-13418, 2021.
  46. Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. arXiv preprint arXiv:2107.02729, 2021. · arXiv
  47. Dipendra Misra, Qinghua Liu, Chi Jin, and John Langford. Provable rich observation reinforcement learning with combinatorial latent states. In International Conference on Learning Representations, 2021.
  48. Sammie Katt, Frans A Oliehoek, and Christopher Amato. Bayesian reinforcement learning in factored pomdps. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 7-15, 2019.
  49. Shagun Sodhani, Franziska Meier, Joelle Pineau, and Amy Zhang. Block contextual mdps for continual learning. In Learning for Dynamics and Control Conference, pages 608-623.
  50. PMLR, 2022.
  51. Beining Han, Chongyi Zheng, Harris Chan, Keiran Paster, Michael Zhang, and Jimmy Ba. Learning domain invariant representations in goal-conditioned block mdps. Advances in Neural Information Processing Systems, 34:764-776, 2021.
  52. Carlos Guestrin, Daphne Koller, Chris Gearhart, and Neal Kanodia. Generalizing plans to new environments in relational mdps. In Proceedings of the 18th international joint conference on Artificial intelligence, pages 1003-1010, 2003.
  53. Craig Boutilier, Raymond Reiter, and Bob Price. Symbolic dynamic programming for first-order mdps. In IJCAI, volume 1, pages 690-700, 2001.
  54. Carlos Diuk, Andre Cohen, and Michael L Littman. An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th international conference on Machine learning, pages 240-247, 2008.
  55. Pashootan Vaezipoor, Andrew C Li, Rodrigo A Toro Icarte, and Sheila A Mcilraith. Ltl2action: Generalizing ltl instructions for multi-task rl. In International Conference on Machine Learning, pages 10497-10508. PMLR, 2021.
  56. Yuqian Jiang, Suda Bharadwaj, Bo Wu, Rishi Shah, Ufuk Topcu, and Peter Stone. Temporal- logic-based reward shaping for continuing reinforcement learning tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7995-8003, 2021a.
  57. Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip S Thomas. Learning action representations for reinforcement learning. arXiv preprint arXiv:1902.00183, 2019. · arXiv
  58. Eric D Langlois and Tom Everitt. How rl agents behave when their actions are modified. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11586-11594, 2021.
  59. Ayush Jain, Norio Kosaka, Kyung-Min Kim, and Joseph J Lim. Know your action set: Learning action relations for reinforcement learning. In International Conference on Learning Representations, 2021.
  60. Brandon Trabucco, Mariano Phielipp, and Glen Berseth. Anymorph: Learning transferable polices by inferring agent morphology. In International Conference on Machine Learning, pages 21677-21691. PMLR, 2022.
  61. Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block mdps. arXiv preprint arXiv:2003.06016, 2020. · arXiv
  62. Rich Caruana. Multitask learning. Machine Learning, 28(1):41-75, 1997. 10.1023/A: 1007379606734. URL http://dx.doi.org/10.1023/A:1007379606734. · DOI · Paper page
  63. Zoltán Gábor, Zsolt Kalmár, and Csaba Szepesvári. Multi-criteria reinforcement learning. In ICML, volume 98, pages 197-205, 1998.
  64. Robert M French. Using semi-distributed representations to overcome catastrophic forgetting in connectionist networks. In Proceedings of the 13th annual cognitive science society conference, volume 1, pages 173-178, 1991.
  65. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782, 2020a. · arXiv
  66. Hui Li, Xuejun Liao, and Lawrence Carin. Multi-task reinforcement learning in partially observable stochastic environments. Journal of Machine Learning Research, 10(May): 1131-1186, 2009.
  67. Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
  68. Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, volume 2. Univ. of Texas, 1992.
  69. Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763, 2016. · arXiv
  70. Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016. · arXiv
  71. Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400, 2017. · arXiv
  72. Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. arXiv preprint arXiv:1707.03141, 2017. · arXiv
  73. Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2018. · arXiv
  74. Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence review, 18(2):77-95, 2002.
  75. Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. arXiv preprint arXiv:2004.05439, 2020. · arXiv
  76. Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211, 1999.
  77. Emmanuel Hadoux, Aurélie Beynier, and Paul Weng. Solving hidden-semi-markov-mode markov decision problems. In International Conference on Scalable Uncertainty Management, pages 176-189. Springer, 2014a.
  78. Erwan Lecarpentier and Emmanuel Rachelson. Non-stationary markov decision processes, a worst-case approach using model-based reinforcement learning. In Advances in Neural Information Processing Systems, pages 7214-7223, 2019.
  79. Wenhao Li, Xiangfeng Wang, Bo Jin, Junjie Sheng, and Hongyuan Zha. Dealing with non-stationarity in marl via trust-region decomposition. In International Conference on Learning Representations, 2021.
  80. Sindhu Padakandla, Shalabh Bhatnagar, et al. Reinforcement learning in non-stationary environments. arXiv preprint arXiv:1905.03970, 2019. · arXiv
  81. Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for non- stationary markov decision processes: The blessing of (more) optimism. In International Conference on Machine Learning, pages 1843-1854. PMLR, 2020.
  82. Omar Darwiche Domingues, Pierre Ménard, Matteo Pirotta, Emilie Kaufmann, and Michal Valko. A kernel-based approach to non-stationary reinforcement learning in metric spaces. arXiv preprint arXiv:2007.05078, 2020. · arXiv
  83. Jürgen Schmidhuber. Curious model-building control systems. In Proc. international joint conference on neural networks, pages 1458-1463, 1991.
  84. Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh. Intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pages 1281-1288, 2005.
  85. Satinder Singh, Richard L Lewis, and Andrew G Barto. Where do rewards come from. In Proceedings of the annual conference of the cognitive science society, pages 2601-2606. Cognitive Science Society, 2009.
  86. Andrew G Barto. Intrinsic motivation and reinforcement learning. In Intrinsically motivated learning in natural and artificial systems, pages 17-47. Springer, 2013.
  87. Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforce- ment learning. arXiv preprint arXiv:1703.01732, 2017. · arXiv
  88. Niels Justesen and Sebastian Risi. Automated curriculum learning by rewarding temporally rare events. In 2018 IEEE Conference on Computational Intelligence and Games (CIG), pages 1-8. IEEE, 2018.
  89. Anirudh Srinivasan, Dzmitry Bahdanau, Maxime Chevalier-Boisvert, and Yoshua Bengio. Automated curriculum generation for policy gradients from demonstrations. arXiv preprint arXiv:1912.00444, 2019. · arXiv
  90. Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Auto- matic curriculum learning for deep rl: A short survey. arXiv preprint arXiv:2003.04664, 2020. · arXiv
  91. Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753, 2019. · arXiv
  92. Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. Advances in Neural Information Processing Systems, 34:5402-5415, 2021.
  93. Jiayu Chen, Yuanxin Zhang, Yuanfan Xu, Huimin Ma, Huazhong Yang, Jiaming Song, Yu Wang, and Yi Wu. Variational automatic curriculum learning for sparse-reward cooperative multi-agent problems. Advances in Neural Information Processing Systems, 34:9681-9693, 2021.
  94. Suyoung Lee and Sae-Young Chung. Improving generalization in meta-rl with imaginary tasks from latent dynamics mixture. Advances in Neural Information Processing Systems, 34:27222-27235, 2021.
  95. Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515-1528, 2018.
  96. Sébastien Racaniere, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap. Automated curricula through setter-solver interactions. arXiv preprint arXiv:1909.12892, 2019. · arXiv
  97. Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Au- tonomous reinforcement learning via subgoal curricula. Advances in Neural Information Processing Systems, 34:18474-18486, 2021.
  98. Pascal Klink, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Boosted curriculum reinforce- ment learning. In International Conference on Learning Representations, 2021.
  99. Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34:1884-1897, 2021b.
  100. Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv:1703.05407, 2017. · arXiv
  101. Stephane Doncieux, Nicolas Bredeche, Léni Le Goff, Benoît Girard, Alexandre Coninx, Olivier Sigaud, Mehdi Khamassi, Natalia Díaz-Rodríguez, David Filliat, Timothy Hospedales, et al. Dream architecture: a developmental approach to open-ended learning in robotics. arXiv preprint arXiv:2005.06223, 2020. · arXiv
  102. Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 122-130. International Foundation for Autonomous Agents and Multiagent Systems, 2018a.
  103. Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric P Xing, and Shimon Whiteson. Dice: The infinitely differentiable monte-carlo estimator. arXiv preprint arXiv:1802.05098, 2018b. · arXiv
  104. Dong Ki Kim, Miao Liu, Matthew D Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, Gerald Tesauro, and Jonathan How. A policy gradient algorithm for learning to learn in multiagent reinforcement learning. In International Conference on Machine Learning, pages 5541-5550. PMLR, 2021a.
  105. Diana Borsa, Thore Graepel, and John Shawe-Taylor. Learning shared representations in multi-task reinforcement learning. arXiv preprint arXiv:1603.02041, 2016. · arXiv
  106. Daniele Calandriello, Alessandro Lazaric, and Marcello Restelli. Sparse multi-task reinforce- ment learning. In Advances in Neural Information Processing Systems, pages 819-827, 2014.
  107. Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016. · arXiv
  108. Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The Benefit of Multitask Representation Learning. JMLR, page 32, 2016.
  109. Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowledge in multi-task deep reinforcement learning. In International Conference on Learning Representations, 2020. URL https://openreview.net/ forum?id=rkgpv2VFvr.
  110. Guanya Shi, Kamyar Azizzadenesheli, Michael O'Connell, Soon-Jo Chung, and Yisong Yue. Meta-adaptive nonlinear control: Theory and algorithms. Advances in Neural Information Processing Systems, 34:10013-10025, 2021.
  111. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836, 2020b.
  112. Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878-18890, 2021a.
  113. Brian Cheung, Alexander Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. In Advances in Neural Information Processing Systems, pages 10868-10877, 2019.
  114. Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. arXiv preprint arXiv:2006.14769, 2020. · arXiv
  115. Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541. ACM, 2006.
  116. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. · arXiv
  117. Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295, 2015. · arXiv
  118. Zhizhong Li and Derek Hoiem. Learning without forgetting. In European Conference on Computer Vision, pages 614-629. Springer, 2016.
  119. Matthew Riemer, Elham Khabiri, and Richard Goodwin. Representation stability as a regu- larizer for improved text analytics transfer learning. arXiv preprint arXiv:1704.03617, 2016. · arXiv
  120. Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018. · arXiv
  121. Jonathan Schwarz, Jelena Luketina, Wojciech M Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. arXiv preprint arXiv:1805.06370, 2018. · arXiv
  122. Glen Berseth, Cheng Xie, Paul Cernek, and Michiel Van de Panne. Progressive rein- forcement learning with distillation for multi-skilled motion control. arXiv preprint arXiv:1802.04765, 2018. · arXiv
  123. Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Policy consolidation for continual reinforcement learning. arXiv preprint arXiv:1902.00255, 2019. · arXiv
  124. René Traoré, Hugo Caselles-Dupré, Timothée Lesort, Te Sun, Guanghang Cai, Natalia Díaz-Rodríguez, and David Filliat. Discorl: Continual reinforcement learning via policy distillation. arXiv preprint arXiv:1907.05855, 2019. · arXiv
  125. Dhruva Tirumala, Hyeonwoo Noh, Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascanu, Yee Whye Teh, and Nicolas Heess. Exploiting hierarchy for learning and transfer in kl-regularized rl. arXiv preprint arXiv:1903.07438, 2019. · arXiv
  126. Maximilian Igl, Gregory Farquhar, Jelena Luketina, JW Böhmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In 9th International Conference on Learning Representations, 2021.
  127. Qingfeng Lan, Yangchen Pan, Jun Luo, and A Rupam Mahmood. Memory-efficient rein- forcement learning with knowledge consolidation. arXiv preprint arXiv:2205.10868, 2022. · arXiv
  128. Tiantian Zhang, Xueqian Wang, Bin Liang, and Bo Yuan. Catastrophic interference in reinforcement learning: A solution based on context division and knowledge distillation. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  129. Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293-321, 1992.
  130. Yangchen Pan, Muhammad Zaheer, Adam White, Andrew Patterson, and Martha White. Organizing experience: a deeper look at replay mechanisms for sample-based planning in continuous state domains. arXiv preprint arXiv:1806.04624, 2018. · arXiv
  131. Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160-163, 1991.
  132. David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
  133. David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, pages 348-358, 2019.
  134. Youngmin Oh, Jinwoo Shin, Eunho Yang, and Sung Ju Hwang. Model-augmented prioritized experience replay. In International Conference on Learning Representations, 2021.
  135. Christian Henning, Maria Cervera, Francesco D'Angelo, Johannes Von Oswald, Regina Traber, Benjamin Ehret, Seijin Kobayashi, Benjamin F Grewe, and João Sacramento. Posterior meta-replay for continual learning. Advances in Neural Information Processing Systems, 34:14135-14149, 2021.
  136. Xu-Hui Liu, Zhenghai Xue, Jingcheng Pang, Shengyi Jiang, Feng Xu, and Yang Yu. Regret minimization experience replay in off-policy reinforcement learning. Advances in Neural Information Processing Systems, 34:17604-17615, 2021b.
  137. James Queeney, Yannis Paschalidis, and Christos G Cassandras. Generalized proximal policy optimization with sample reuse. Advances in Neural Information Processing Systems, 34:11909-11919, 2021.
  138. Yash Chandak, Scott Niekum, Bruno da Silva, Erik Learned-Miller, Emma Brunskill, and Philip S Thomas. Universal off-policy evaluation. Advances in Neural Information Processing Systems, 34:27475-27490, 2021.
  139. Andrew Lampinen, Stephanie Chan, Andrea Banino, and Felix Hill. Towards mental time travel: a hierarchical memory for reinforcement learning agents. Advances in Neural Information Processing Systems, 34:28182-28195, 2021.
  140. David Venuto, Elaine Lau, Doina Precup, and Ofir Nachum. Policy gradients incorporating the future. arXiv preprint arXiv:2108.02096, 2021. · arXiv
  141. Pierre Liotet, Francesco Vidaich, Alberto Maria Metelli, and Marcello Restelli. Lifelong hyper- policy optimization with multiple importance sampling regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7525-7533, 2022.
  142. Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):123-146, 1995.
  143. Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony Robins. Pseudo-rehearsal: Achieving deep reinforcement learning without catastrophic forgetting. arXiv preprint arXiv:1812.02464, 2018. · arXiv
  144. Zachary Daniels, Aswin Raghavan, Jesse Hostetler, Abrar Rahman, Indranil Sur, Michael Piacentino, and Ajay Divakaran. Model-free generative replay for lifelong reinforcement learning: Application to starcraft-2. arXiv preprint arXiv:2208.05056, 2022. · arXiv
  145. Matthew Riemer, Tim Klinger, Djallel Bouneffouf, and Michele Franceschini. Scalable recollections for continual lifelong learning. arXiv preprint arXiv:1711.06761, 2017. · arXiv
  146. Lucas Caccia, Eugene Belilovsky, Massimo Caccia, and Joelle Pineau. Online learned contin- ual compression with stacked quantization module. arXiv preprint arXiv:1911.08019, 2019. · arXiv
  147. Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Continual reinforcement learning with complex synapses. arXiv preprint arXiv:1802.07239, 2018. · arXiv
  148. Sebastian Thrun and Joseph O'Sullivan. Discovering structure in multiple learning tasks: The tc algorithm. In ICML, volume 96, pages 489-497, 1996.
  149. Ronald Parr and Stuart J Russell. Reinforcement learning with hierarchies of machines. In Advances in neural information processing systems, pages 1043-1049, 1998.
  150. Sebastian Thrun and Anton Schwartz. Finding structure in reinforcement learning. In Advances in neural information processing systems, pages 385-392, 1995.
  151. Satinder Pal Singh. Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning, 8(3-4):323-339, 1992.
  152. Kenji Doya, Kazuyuki Samejima, Ken-ichi Katagiri, and Mitsuo Kawato. Multiple model- based reinforcement learning. Neural computation, 14(6):1347-1369, 2002.
  153. Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2169-2176. IEEE, 2017.
  154. Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. arXiv preprint arXiv:1710.09767, 2017. · arXiv
  155. Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734, 2017. · arXiv
  156. Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selec- tion of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. · arXiv
  157. Elliot Meyerson and Risto Miikkulainen. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. arXiv preprint arXiv:1711.00108, 2017. · arXiv
  158. Louis Kirsch, Julius Kunze, and David Barber. Modular networks: learning to decompose neural computation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 2414-2423, 2018.
  159. Prajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models. ICLR, 2018.
  160. Michael B Chang, Abhishek Gupta, Sergey Levine, and Thomas L Griffiths. Automatically composing representation transformations as a means for generalization. arXiv preprint arXiv:1807.04640, 2018. · arXiv
  161. Jason Liang, Elliot Meyerson, and Risto Miikkulainen. Evolutionary architecture search for deep multitask networks. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 466-473, 2018.
  162. Ferran Alet, Tomás Lozano-Pérez, and Leslie P Kaelbling. Modular meta-learning. arXiv preprint arXiv:1806.10166, 2018. · arXiv
  163. Ignacio Cases, Clemens Rosenbaum, Matthew Riemer, Atticus Geiger, Tim Klinger, Alex Tamkin, Olivia Li, Sandhini Agarwal, Joshua D Greene, Dan Jurafsky, et al. Recursive routing networks: Learning to compose modules for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3631-3648, 2019.
  164. Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. arXiv preprint arXiv:2003.13661, 2020. · arXiv
  165. Seungwon Lee, Sima Behpour, and Eric Eaton. Sharing less is more: Lifelong learning in deep networks with selective layer transfer. In International Conference on Machine Learning, pages 6065-6075. PMLR, 2021.
  166. Wei-Cheng Tseng, Jin-Siang Lin, Yao-Min Feng, and Min Sun. Toward robust long range policy transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9958-9966, 2021.
  167. Jorge A Mendez and Eric Eaton. How to reuse and compose knowledge for a lifetime of tasks: A survey on continual learning and functional composition. arXiv preprint arXiv:2207.07730, 2022. · arXiv
  168. Jorge A Mendez, Harm van Seijen, and Eric Eaton. Modular lifelong reinforcement learning via neural composition. arXiv preprint arXiv:2207.00429, 2022. · arXiv
  169. Jean-Baptiste Gaya, Thang Doan, Lucas Caccia, Laure Soulier, Ludovic Denoyer, and Roberta Raileanu. Building a subspace of policies for scalable continual learning. In Decision Awareness in Reinforcement Learning Workshop at ICML 2022, 2022.
  170. Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016. · arXiv
  171. Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model architecture search through hypernetworks. arXiv preprint arXiv:1708.05344, 2017. · arXiv
  172. Ramakanth Pasunuru and Mohit Bansal. Continual and multi-task architecture search. arXiv preprint arXiv:1906.05226, 2019. · arXiv
  173. Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Advances in Neural Information Processing Systems, pages 899-908, 2018.
  174. Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing net- works and the challenges of modular and compositional computation. arXiv preprint arXiv:1904.12774, 2019. · arXiv
  175. David Abel, Dilip Arumugam, Lucas Lehnert, and Michael L. Littman. State abstractions for lifelong reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2018.
  176. Amy Zhang, Harsh Satija, and Joelle Pineau. Decoupling dynamics and reward for transfer learning. arXiv preprint arXiv:1804.10689, 2018b. · arXiv
  177. Vincent François-Lavet, Yoshua Bengio, Doina Precup, and Joelle Pineau. Combined reinforcement learning via abstract representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3582-3589, 2019.
  178. Amy Zhang, Zachary C Lipton, Luis Pineda, Kamyar Azizzadenesheli, Anima Anandku- mar, Laurent Itti, Joelle Pineau, and Tommaso Furlanello. Learning causal state representations of partially observable environments. arXiv preprint arXiv:1906.10437, 2019. · arXiv
  179. Philippe Hansen-Estruch, Amy Zhang, Ashvin Nair, Patrick Yin, and Sergey Levine. Bisim- ulation makes analogies in goal-conditioned reinforcement learning. arXiv preprint arXiv:2204.13060, 2022. · arXiv
  180. C Chace Ashcraft, Benjamin Stoler, Chigozie Ewulum, and Susama Agarwala. Structural sim- ilarity for improved transfer in reinforcement learning. arXiv preprint arXiv:2207.13813, 2022. · arXiv
  181. Cameron Allen, Neev Parikh, Omer Gottesman, and George Konidaris. Learning markov state abstractions for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:8229-8241, 2021.
  182. Christopher Grimm, André Barreto, Satinder Singh, and David Silver. The value equivalence principle for model-based reinforcement learning. Advances in Neural Information Processing Systems, 33:5541-5552, 2020.
  183. Samuel Sokota, Caleb Y Ho, Zaheen Ahmad, and J Zico Kolter. Monte carlo tree search with iteratively refining state abstractions. Advances in Neural Information Processing Systems, 34:18698-18709, 2021.
  184. Brandon Cui, Yinlam Chow, and Mohammad Ghavamzadeh. Control-aware representations for model-based reinforcement learning. In International Conference on Learning Representations, 2021.
  185. Aidan Curtis, Tom Silver, Joshua B Tenenbaum, Tomás Lozano-Pérez, and Leslie Kaelbling. Discovering state and action abstractions for generalized task and motion planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 5377-5384, 2022.
  186. Marwa Abdulhai, Dong Ki Kim, Matthew Riemer, Miao Liu, Gerald Tesauro, and Jonathan P How. Context-specific representation abstraction for deep option learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 36, pages 5959-5967, 2022.
  187. Milos Hauskrecht, Nicolas Meuleau, Leslie Pack Kaelbling, Thomas Dean, and Craig Boutilier. Hierarchical solution of markov decision processes using macro-actions. In Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence. Morgan Kaufmann Publishers Inc., 1998.
  188. Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty- First AAAI Conference on Artificial Intelligence, 2017.
  189. Marios C Machado, Marc G Bellemare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2295-2304. JMLR. org, 2017a.
  190. Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. arXiv preprint arXiv:1710.11089, 2017b. · arXiv
  191. Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not an option: Learning options with a deliberation cost. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
  192. Khimya Khetarpal, Martin Klissarov, Maxime Chevalier-Boisvert, Pierre-Luc Bacon, and Doina Precup. Options of interest: Temporal abstraction with interest functions. arXiv preprint arXiv:2001.00271, 2020a. · arXiv
  193. Matthew Riemer, Ignacio Cases, Clemens Rosenbaum, Miao Liu, and Gerald Tesauro. On the role of weight sharing during deep option learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5519-5526, 2020.
  194. Martin Klissarov and Doina Precup. Flexible option learning. Advances in Neural Information Processing Systems, 34:4632-4646, 2021.
  195. Emma Brunskill and Lihong Li. Pac-inspired option discovery in lifelong reinforcement learning. In International Conference on Machine Learning, pages 316-324, 2014.
  196. Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv:1807.10299, 2018. · arXiv
  197. Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. NIPS, 2018.
  198. Maximilian Igl, Andrew Gambardella, Jinke He, Nantas Nardelli, N Siddharth, Wendelin Böhmer, and Shimon Whiteson. Multitask soft option learning. arXiv preprint arXiv:1904.01033, 2019. · arXiv
  199. Daniel J Mankowitz, Timothy A Mann, and Shie Mannor. Adaptive skills adaptive partitions (asap). In Advances in Neural Information Processing Systems, pages 1588-1596, 2016.
  200. Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018. · arXiv

200 of 426 shown

Cited by

Sorted by citations

  • A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

    Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193 · see citing works

    “Khetarpal et al. (2020) motivated and surveyed continual reinforcement learning (CRL), which is closely related to ZSG in RL as both settings require adaptation to unseen tasks or environments; however, they explicitly do not discuss the zero-shot setting that is the concern of this paper (for more…”
    BackgroundInfluential

Authors
Khetarpal, K., Riemer, M., Rish, I., & Precup, D.
Affiliations
  • McGill University (CA)
  • Mila - Quebec Artificial Intelligence Institute (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
75
Pages
1401-1476
ISSN
1076-9757 (ISSN-L)
DOI
10.48550/arxiv.2012.13490
arXiv
2012.13490
OpenAlex
W3117215073
Cited by
227
References
1
Citations by year
2026: 38 · 2025: 54 · 2024: 72 · 2023: 29 · 2022: 11 · 2021: 23
FWCI
13.60
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Artificial Intelligence Research (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Canadian Institute for Advanced Research
  • Canada Excellence Research Chairs, Government of Canada
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
Keywords
continual reinforcement learning, lifelong learning, incremental learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review