CoChat
Sign Up
PreprintCited by 4Domain Adaptation and Few-Shot LearningReinforcement Learning in RoboticsMachine Learning and Algorithms

The DOI matches its DataCite record; the references it cites are not checked yet.

Continual Learning as Computationally Constrained Reinforcement Learning

Open access
Kumar, Saurabh, Marklund, Henrik, Rao, Ashish, Zhu, Yifan, Jeon, Hong Jun, Liu, Yueyang, Van Roy, Benjamin·2023·arXiv
DOI 10.48550/arxiv.2307.04345arXiv 2307.04345
Open this paper in CoChat — free accountOpen PDF

Abstract

An agent that efficiently accumulates knowledge to develop increasingly sophisticated skills over a long lifetime could advance the frontier of artificial intelligence capabilities. The design of such agents, which remains a long-standing challenge of artificial intelligence, is addressed by the subject of continual learning. This monograph clarifies and formalizes concepts of continual learning, introducing a framework and set of tools to stimulate further research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.04345

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

References 27 · Cited by 4

Research works printed but not linked: 108

  1. Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pages 10-4, 2019.
  2. Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019.
  3. Dilip Arumugam and Benjamin Van Roy. Deciding what to learn: A rate-distortion approach. In International Conference on Machine Learning, pages 373-382. PMLR, 2021a.
  4. Dilip Arumugam and Benjamin Van Roy. The value of information when deciding what to learn. Advances in neural information processing systems, 34:9816-9827, 2021b.
  5. Dylan R Ashley, Sina Ghiassian, and Richard S Sutton. Does the Adam optimizer exacerbate catastrophic forget- ting? arXiv preprint arXiv:2102.07686, 2021. · arXiv
  6. Peter Auer, Pratik Gajane, and Ronald Ortner. Adaptively tracking the best bandit arm with an unknown number of distribution changes. In Conference on Learning Theory, pages 138-158. PMLR, 2019.
  7. Dheeraj Baby, Saurabh Garg, Tzu-Ching Yen, Sivaraman Balakrishnan, Zachary Lipton, and Yu-Xiang Wang. On- line label shift: Optimal dynamic regret meets practical algorithms. Advances in Neural Information Processing Systems, 36, 2024.
  8. Benedikt Bagus, Alexander Gepperth, and Timothée Lesort. Beyond supervised continual learning: a review, 2022.
  9. Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non-stationary rewards. Advances in neural information processing systems, 27, 2014.
  10. Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations research, 63(5): 1227-1244, 2015.
  11. Omar Besbes, Yonatan Gur, and Assaf Zeevi. Optimal exploration-exploitation in a multi-armed-bandit problem with non-stationary rewards. Stochastic Systems, 9(4):319-337, 2019.
  12. Lilian Besson and Emilie Kaufmann. The generalized likelihood ratio test meets KLUCB: an improved algorithm for piece-wise non-stationary bandits. Proceedings of Machine Learning Research vol XX, 1:35, 2019.
  13. Ilija Bogunovic, Jonathan Scarlett, and Volkan Cevher. Time-varying gaussian process bandit optimization. In Artificial Intelligence and Statistics, pages 314-323. PMLR, 2016.
  14. Leo Breiman. The individual ergodic theorem of information theory. The Annals of Mathematical Statistics, 28(3): 809-811, 1957.
  15. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
  16. Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33: 15920-15930, 2020.
  17. Zhipeng Cai, Ozan Sener, and Vladlen Koltun. Online continual learning with natural distribution shifts: An empirical study with visual data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8281-8290, 2021.
  18. Annie Chen, Archit Sharma, Sergey Levine, and Chelsea Finn. You only live once: Single-life reinforcement learning. Advances in Neural Information Processing Systems, 35:14784-14797, 2022.
  19. Qinyi Chen, Negin Golrezaei, and Djallel Bouneffouf. Non-stationary bandits with auto-regressive temporal depen- dency. Advances in Neural Information Processing Systems, 36:7895-7929, 2023.
  20. Yifang Chen, Chung-Wei Lee, Haipeng Luo, and Chen-Yu Wei. A new algorithm for non-stationary contextual bandits: Efficient, optimal and parameter-free. In Conference on Learning Theory, pages 696-726. PMLR, 2019.
  21. Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Learning to optimize under non-stationarity. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1079-1087. PMLR, 2019.
  22. Aristotelis Chrysakis and Marie-Francine Moens. Online continual learning from imbalanced data. In International Conference on Machine Learning, pages 1952-1961. PMLR, 2020.
  23. Thomas M Cover and Joy A Thomas. Elements of Information Theory. John Wiley & Sons, 2 edition, 2006.
  24. Thomas M Cover and Joy A Thomas. Elements of Information Theory. John Wiley & Sons, 2012.
  25. Amit Daniely, Alon Gonen, and Shai Shalev-Shwartz. Strongly adaptive online learning. In International Conference on Machine Learning, pages 1405-1411. PMLR, 2015.
  26. Mayank Daswani, Peter Sunehag, and Marcus Hutter. Q-learning for history-based reinforcement learning. In Asian Conference on Machine Learning, pages 213-228. PMLR, 2013.
  27. Mayank Daswani, Peter Sunehag, Marcus Hutter, et al. Feature reinforcement learning: state of the art. In Sequential decision-making with big data: papers from the AAAI-14 workshop. Association for the Advancement of Artificial Intelligence, 2014.
  28. Bruno de Finetti. Funzione caratteristica di un fenomeno aleatorio. In Atti del Congresso Internazionale dei Matematici: Bologna del 3 al 10 de settembre di 1928, pages 179-190, 1929.
  29. Rommert Dekker and Arie Hordijk. Average, sensitive and blackwell optimal policies in denumerable markov decision chains with unbounded rewards. Mathematics of Operations Research, 13(3):395-420, 1988.
  30. Vektor Dewanto, George Dunn, Ali Eshragh, Marcus Gallagher, and Fred Roosta. Average-reward model-free reinforcement learning: a systematic review and literature mapping. arXiv preprint arXiv:2010.08920, 2020. · arXiv
  31. Persi Diaconis and David Freedman. de Finetti's theorem for Markov chains. The Annals of Probability, pages 115-130, 1980.
  32. Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325, 2021. · arXiv
  33. Omar Darwiche Domingues, Pierre Ménard, Matteo Pirotta, Emilie Kaufmann, and Michal Valko. A kernel-based approach to non-stationary reinforcement learning in metric spaces. In International Conference on Artificial Intelligence and Statistics, pages 3538-3546. PMLR, 2021.
  34. Shi Dong, Benjamin Van Roy, and Zhengyuan Zhou. Simple agent, complex environment: Efficient reinforcement learning with agent states. Journal of Machine Learning Research, 23(255):1-54, 2022.
  35. Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl 2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016. · arXiv
  36. Yingjie Fei, Zhuoran Yang, Zhaoran Wang, and Qiaomin Xie. Dynamic regret of policy optimization in non- stationary environments. Advances in Neural Information Processing Systems, 33:6743-6754, 2020.
  37. Enrico Fini, Stéphane Lathuiliere, Enver Sangineto, Moin Nabi, and Elisa Ricci. Online continual learning under extreme memory constraints. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII 16, pages 720-735. Springer, 2020.
  38. Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, and Satinder Singh. Bootstrapped meta-learning. arXiv preprint arXiv:2109.04504, 2021. · arXiv
  39. Yaniv Fogel and Meir Feder. On the problem of on-line learning with log-loss. In 2017 IEEE International Symposium on Information Theory (ISIT), pages 2995-2999. IEEE, 2017.
  40. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2, 2023. · arXiv
  41. Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for non-stationary bandit problems. arXiv preprint arXiv:0805.3415, 2008. · arXiv
  42. Gourab Ghatak. A change-detection-based Thompson sampling framework for non-stationary bandits. IEEE Transactions on Computers, 70(10):1670-1676, 2021. doi: 10.1109/TC.2020.3022634. · DOI
  43. Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip HS Torr, and Bernard Ghanem. Real-time evaluation in online continual learning: A new paradigm. arXiv preprint arXiv:2302.01047, 2023. · arXiv
  44. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013. · arXiv
  45. Neha Gupta, Ole-Christoffer Granmo, and Ashok Agrawala. Thompson sampling for dynamic multi-armed bandits. In 2011 10th International Conference on Machine Learning and Applications and Workshops, volume 1, pages 484-489. IEEE, 2011.
  46. Raia Hadsell, Dushyant Rao, Andrei A. Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 24(12):1028-1040, 2020.
  47. Hasan Abed Al Kader Hammoud, Ameya Prabhu, Ser-Nam Lim, Philip HS Torr, Adel Bibi, and Bernard Ghanem. Rapid adaptation in online continual learning: Are we evaluating it right? arXiv preprint arXiv:2305.09275, 2023. · arXiv
  48. Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. · arXiv
  49. Hexiang Hu, Ozan Sener, Fei Sha, and Vladlen Koltun. Drinking from a firehose: Continual learning with web-scale natural language. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  50. Marcus Hutter. Universal algorithmic intelligence: A mathematical top→down approach. In B. Goertzel and C. Pennachin, editors, Artificial General Intelligence, Cognitive Technologies, pages 227-290. Springer, Berlin, 2007.
  51. Hong Jun Jeon and Benjamin Van Roy. Information-theoretic foundations for machine learning. arXiv preprint arXiv:2407.12288, 2024. · arXiv
  52. Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Transactions of the ASME- Journal of Basic Engineering, 82(Series D):35-45, 1960.
  53. Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 75:1401-1476, 2022.
  54. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, and Demis Hassibis. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-3526, 2017.
  55. Levente Kocsis and Csaba Szepesvári. Discounted UCB. In 2nd PASCAL Challenges Workshop, volume 2, pages 51-134, 2006.
  56. Hyunseo Koh, Minhyuk Seo, Jihwan Bang, Hwanjun Song, Deokki Hong, Seulki Park, Jung-Woo Ha, and Jonghyun Choi. Online boundary-free continual learning by scheduled data prior. In The Eleventh International Conference on Learning Representations, 2023.
  57. Andre Nikolaevich Kolmogorov. Foundations of the theory of probability: Second English Edition. Courier Dover Publications, 2018.
  58. Julia Kuhn and Yoni Nazarathy. Wireless channel selection with reward-observing restless multi-armed bandits. Chapter to appear in "Markov Decision Processes in Practice", Editors: R. Boucherie and N. van Dijk, 2015.
  59. Julia Kuhn, Michel Mandjes, and Yoni Nazarathy. Exploration vs exploitation with partially observable gaussian autoregressive arms. EAI Endorsed Transactions on Self-Adaptive Systems, 1(4), 2 2015. doi: 10.4108/icst. valuetools.2014.258207. · DOI
  60. Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
  61. Timothée Lesort. Continual learning: Tackling catastrophic forgetting in deep neural networks with replay pro- cesses. arXiv preprint arXiv:2007.00487, 2020. · arXiv
  62. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in neural information processing systems, 33:9459-9474, 2020.
  63. Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learning on real-world imagery. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  64. Yueyang Liu, Benjamin Van Roy, and Kuang Xu. A definition of non-stationary bandits. arXiv preprint arXiv:2302.12202, 2023a. · arXiv
  65. Yueyang Liu, Benjamin Van Roy, and Kuang Xu. Nonstationary bandit learning via predictive sampling. In International Conference on Artificial Intelligence and Statistics, pages 6215-6244.
  66. PMLR, 2023b.
  67. Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, Ian Osband, and Zheng Wen. Rein- forcement learning, bit by bit. Foundations and Trends® in Machine Learning, 16(6):733-865, 2023.
  68. Haipeng Luo, Chen-Yu Wei, Alekh Agarwal, and John Langford. Efficient contextual bandits in non-stationary worlds. In Conference On Learning Theory, pages 1739-1776. PMLR, 2018.
  69. Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. arXiv preprint arXiv:2303.01486, 2023. · arXiv
  70. R Andrew McCallum. Instance-based utile distinctions for reinforcement learning with hidden state. In Machine Learning Proceedings 1995, pages 387-395. Elsevier, 1995.
  71. Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. volume 24 of Psychology of Learning and Motivation, pages 109-165. Academic Press, 1989.
  72. Brockway McMillan. The basic theorems of information theory. The Annals of mathematical statistics, pages 196-219, 1953.
  73. Joseph Mellor and Jonathan Shapiro. Thompson sampling in switching environments with Bayesian online change detection. In Carlos M. Carvalho and Pradeep Ravikumar, editors, Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, volume 31 of Proceedings of Machine Learning Research, pages 442-450, Scottsdale, Arizona, USA, 29 Apr-01 May 2013. PMLR. URL https://proceedings.mlr. press/v31/mellor13a.html.
  74. Sean Meyn. Control systems and reinforcement learning. Cambridge University Press, 2022.
  75. Seungki Min and Daniel Russo. An information-theoretic analysis of nonstationary bandit learning. In International Conference on Machine Learning, pages 24831-24849. PMLR, 2023.
  76. Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Fara- jtabar. Wide neural networks forget less catastrophically. In International Conference on Machine Learning, pages 15699-15717. PMLR, 2022.
  77. Jose G Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern recognition, 45(1):521-530, 2012.
  78. Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. arXiv preprint arXiv:2305.15555, 2023. · arXiv
  79. Sindhu Padakandla. A survey of reinforcement learning algorithms for dynamically varying environments. ACM Computing Surveys (CSUR), 54(6):1-25, 2021.
  80. Ameya Prabhu, Zhipeng Cai, Puneet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener. Online continual learning without the storage constraint. arXiv preprint arXiv:2305.09253, 2023a. · arXiv
  81. Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? arXiv preprint arXiv:2303.11165, 2023b. · arXiv
  82. Vishnu Raj and Sheetal Kalyani. Taming non-stationary bandits: A Bayesian approach. arXiv preprint arXiv:1707.09727, 2017. · arXiv
  83. Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
  84. Mark B. Ring. Continual Learning in Reinforcement Environments. PhD thesis, University of Texas at Austin, Austin, Texas 78712, August 1994.
  85. Mark B Ring. Toward a formal framework for continual learning. In NIPS workshop on Inductive Transfer, Whistler, Canada, 2005.
  86. Stuart J Russell and Peter Norvig. Artificial intelligence: a modern approach. Pearson, 2016.
  87. Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al. A tutorial on Thompson sampling. Foundations and Trends® in Machine Learning, 11(1):1-96, 2018.
  88. Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948.
  89. Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, and Chelsea Finn. Autonomous reinforcement learning: Formalism and benchmarking. arXiv preprint arXiv:2112.09605, 2021. · arXiv
  90. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Out- rageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. · arXiv
  91. Yanina Shkel, Maxim Raginsky, and Sergio Verdú. Sequential prediction with coded side information under loga- rithmic loss. In Algorithmic Learning Theory, pages 753-769. PMLR, 2018.
  92. Aleksandrs Slivkins and Eli Upfal. Adapting to a changing environment: the Brownian restless bandits. In COLT, pages 343-354, 2008.
  93. Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pages 216-224. Elsevier, 1990.
  94. Richard S. Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In Proceedings of the Tenth National Conference on Artificial Intelligence, page pp. 171-176. MIT Press, 1992.
  95. Richard S Sutton, Anna Koop, and David Silver. On the role of tracking in stationary environments. In Proceedings of the 24th international Conference on Machine Learning, pages 871-878, 2007.
  96. Csaba Szepesvári. Algorithms for reinforcement learning. Synthesis lectures on artificial intelligence and machine learning, 4(1):1-103, 2010.
  97. Ryosuke Takahira, Kumiko Tanaka-Ishii, and Lukasz Debowski. Entropy rate estimates for natural language-a new extrapolation of compressed large-scale corpora. Entropy, 18(10):364, 2016.
  98. William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.
  99. Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. Learning to learn, pages 3-17, 1998.
  100. Francesco Trovo, Stefano Paladino, Marcello Restelli, and Nicola Gatti. Sliding-window Thompson sampling for non-stationary settings. Journal of Artificial Intelligence Research, 68:311-364, 2020.
  101. Tim Van Erven and Wouter M Koolen. Metagrad: Multiple learning rates in online learning. Advances in Neural Information Processing Systems, 29, 2016.
  102. Paolo Viappiani. Thompson sampling for Bayesian bandits with resets. In International Conference on Algorithmic Decision Theory, pages 399-410. Springer, 2013.
  103. Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11 (1):37-57, 1985.
  104. Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487, 2023. · arXiv
  105. Bernard Widrow and Marcian E. Hoff, Jr. Adaptive switching circuits. In IRE WESCON Convention Record, volume 4, pages 96-104, 1960.
  106. Zhenbo Xu, Haimiao Hu, and Liu Liu. Revealing the real-world applicable setting of online continual learning. In 2022 IEEE 24th International Workshop on Multimedia Signal Processing (MMSP), pages 1-5. IEEE, 2022.
  107. Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. arXiv preprint arXiv:2106.01085, 2021. · arXiv
  108. Lijun Zhang, Shiyin Lu, and Zhi-Hua Zhou. Adaptive online learning in dynamic environments. Advances in neural information processing systems, 31, 2018.

Metadata

Authors
Kumar, S., Marklund, H., Rao, A., Zhu, Y., Jeon, H. J., Liu, Y., & Van Roy, B.
Affiliations
  • Stanford University (US)
  • Rice University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
18
Issue
5
Pages
913-1053
ISSN
1935-8237 (ISSN-L)
DOI
10.48550/arxiv.2307.04345
arXiv
2307.04345
OpenAlex
W4413358389
Cited by
4
References
27
Citations by year
2026: 3 · 2024: 1
FWCI
1.21
Type
journal article
Language
en
Open access
Licence
Locations
  • Foundations and Trends® in Machine Learning (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Army Research Office (W911NF2010055)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
continual learning, reinforcement learning, computational resource constraints, catastrophic forgetting, relearning, exploration, auxiliary learning, information constraints, stability-plasticity dilemma

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Continual Learning as Computationally Constrained Reinforcement Learning. https://app.cochat.ai/papers/10.48550/arxiv.2307.04345
The paper
Kumar, S., Marklund, H., Rao, A., Zhu, Y., Jeon, H. J., Liu, Y., & Van Roy, B. (2023). Continual Learning as Computationally Constrained Reinforcement Learning. arXiv. https://doi.org/10.48550/arxiv.2307.04345