CoChat

A Definition of Continual Reinforcement Learning

Open access
Abel, David, Barreto, Andre, Roy, Benjamin Van, Precup, Doina, Hasselt, Hado Van, Singh, Satinder · 2023 · Advances in Neural Information Processing Systems 36

The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index; none of the 13 sources its claims cite is checked for support yet.

Abstract

In a standard view of the reinforcement learning problem, an agent's goal is to efficiently identify a policy that maximizes long-term reward. However, this perspective is based on a restricted view of learning as finding a solution, rather than treating learning as endless adaptation. In contrast, continual reinforcement learning refers to the setting in which the best agents never stop learning. Despite the importance of continual reinforcement learning, the community lacks a simple definition of the problem that highlights its commitments and makes its primary concepts precise and clear. To this end, this paper is dedicated to carefully defining the continual reinforcement learning problem. We formalize the notion of agents that "never stop learning" through a new mathematical language for analyzing and cataloging agents. Using this new language, we define a continual learning agent as one that can be understood as carrying out an implicit search process indefinitely, and continual reinforcement learning as the setting in which the best agents are all continual learning agents. We provide two motivating examples, illustrating that traditional views of multi-task reinforcement learning and continual supervised learning are special cases of our definition. Collectively, these definitions and perspectives formalize many intuitive concepts at the heart of learning, and open new research pathways surrounding continual learning agents.

Overview

CoChat Summary

TL;DR

The paper proposes a formal definition of continual reinforcement learning (CRL) as RL problems in which all optimal agents never stop their implicit search over an agent basis. It introduces the operators *generates* and *reaches* to make this precise, and shows multi-task RL and continual supervised learning as special cases.

Key findings

  • CRL is defined for an RL problem \((e,v,\Lambda)\) with basis \(\Lambda_B \subset \Lambda\) generating \(\Lambda\): it is CRL if every optimal agent \(\lambda^*\) never reaches \(\Lambda_B\) (Definition 4.2). A continual learning agent is one where \(\Lambda_B \vdash_e \{\lambda\}\) and \(\lambda \not\rightsquigarrow_e \Lambda_B\) (Definition 4.1).
  • Theorem 3.1 shows that for any agent-environment pair \((\lambda,e)\), there are infinitely many bases \(\Lambda_B\) such that \(\lambda \notin \Lambda_B\) and \(\Lambda_B \vdash_e \{\lambda\}\); Remark 3.2 says every agent either sometimes or never reaches such a basis.
  • In a switching-MDP example with \(n=10\) underlying MDPs, switch probability \(0.001\), and \(\epsilon=0.15\), continual Q-learning with constant step size \(\alpha=0.1\) outperformed convergent Q-learning that anneals its step size to zero, using average reward with 95% confidence intervals over 250 runs.
  • Theorem 4.1 states no basis element is optimal in CRL and \(\Lambda\) is not minimal; Theorem 4.2 shows *generates* is transitive, not commutative, reflexive, and its decision problem is undecidable; Theorem 4.3 shows *reaches* is not transitive, not commutative, reflexive, and its decision problem is undecidable.

Methods

  • Conceptual/theoretical paper: formalizes agent-environment interfaces, histories, agents as history-based policies, environments, reward, and performance; introduces agent bases, learning rules, and the *generates* and *reaches* operators.
  • Main analysis consists of formal definitions and theorems, with proofs deferred to Appendix B and additional analysis in Appendix C.
  • Two motivating examples: a switching-MDP Q-learning experiment (as above) and a continual supervised learning setting adapted from Mai et al., argued to be a special case of the CRL definition under sufficient distributional shift.

Limitations

  • Paper-stated: the definition is basis-dependent, so an agent may be continual relative to one basis but not another; the authors argue the basis is not arbitrary in practice because resource constraints and design choices restrict it. They also defer much analysis and proofs to the appendix and note further exploration of CRL properties as future work.
  • Paper-stated: they remain agnostic to how reward is aggregated and assume the reward hypothesis, so all relevant goals are captured by a deterministic reward function.
  • Evident from the text: empirical support is limited to one simple switching-MDP Q-learning comparison; no broader benchmark or real-world validation is reported. Key decision problems for the operators are undecidable, and the examples are illustrative rather than exhaustive.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.52202/075280-2192

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 9 Not valid · 11 Not checked · 70 Own results and interpretations

50 more claims

Create a free account to see all 90 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 26
  • It may state two separate claims as one. 14

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 44 references

Research works printed but not linked: 20

  1. David Abel, Yuu Jinnai, Yue Guo, George Konidaris, and Michael L. Littman. Policy and value transfer in lifelong reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2018.
  2. Peter L Bartlett. Learning with a slowly changing distribution. In Proceedings of the Annual Workshop on Computational Learning Theory, 1992.
  3. Emma Brunskill and Lihong Li. PAC-inspired option discovery in lifelong reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2014.
  4. Anthony R. Cassandra, Leslie Pack Kaelbling, and Michael L. Littman. Acting optimally in partially observable stochastic domains. In Proceedings of the AAAI Conference on Artificiall Intelligence, 1994.
  5. Milton Friedman. Essays in positive economics. University of Chicago press, 1953.
  6. Haotian Fu, Shangqun Yu, Michael Littman, and George Konidaris. Model-based lifelong reinforcement learning with Bayesian exploration. Advances in Neural Information Processing Systems, 2022.
  7. Marcus Hutter. Universal artificial intelligence: Sequential decisions based on algorithmic probability. Springer Science & Business Media, 2004.
  8. Tor Lattimore. Theory of general reinforcement learning. PhD thesis, The Australian National University, 2014.
  9. Jan Leike. Nonparametric general reinforcement learning. PhD thesis, The Australian National University, 2016.
  10. Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, Ian Osband, and Zheng Wen. Reinforcement learning, bit by bit. Foundations and Trends in Machine Learning, 16(6):733-865, 2023. ISSN 1935-8237.
  11. Sultan J Majeed. Abstractions of general reinforcement Learning. PhD thesis, The Australian National University, 2021.
  12. Claire Monteleoni and Tommi Jaakkola. Online learning of non-stationary sequences. Advances in Neural Information Processing Systems, 16, 2003.
  13. Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
  14. Mark B Ring. Continual learning in reinforcement environments. PhD thesis, The University of Texas at Austin, 1994.
  15. Mark B Ring. Toward a formal framework for continual learning. In NeurIPS Workshop on Inductive Transfer, 2005.
  16. Paul Ruvolo and Eric Eaton. ELLA: An efficient lifelong learning algorithm. In Proceedings of the International Conference on Machine Learning, 2013.
  17. Daniel L Silver. Machine lifelong learning: Challenges and benefits for artificial general intelligence. In Proceedings of the Conference on Artificial General Intelligence, 2011.
  18. Richard S Sutton. Introduction: The challenge of reinforcement learning. In Reinforcement Learning, pages 1-3. Springer, 1992.
  19. Richard S Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
  20. Sebastian Thrun. Is learning the n-th thing any easier than learning the first? Advances in Neural Information Processing Systems, 1995.

Cited by

See all 66

Metadata

Authors
Abel, D., Barreto, A., Roy, B. V., Precup, D., Hasselt, H. V., & Singh, S.
Venue
Advances in Neural Information Processing Systems 36
Published
2023
Pages
50377-50407
ISSN
1049-5258 (ISSN-L)
OpenAlex
W7133245600
Cited by
36
References
10
Citations by year
2026: 15 · 2025: 7 · 2024: 13 · 2023: 1
FWCI
6.45
Type
conference paper
Language
en
Open access
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
continual reinforcement learning, definitions, lifelong learning, online learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.