A Definition of Continual Reinforcement Learning
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index; none of the 13 sources its claims cite is checked for support yet.
Abstract
In a standard view of the reinforcement learning problem, an agent's goal is to efficiently identify a policy that maximizes long-term reward. However, this perspective is based on a restricted view of learning as finding a solution, rather than treating learning as endless adaptation. In contrast, continual reinforcement learning refers to the setting in which the best agents never stop learning. Despite the importance of continual reinforcement learning, the community lacks a simple definition of the problem that highlights its commitments and makes its primary concepts precise and clear. To this end, this paper is dedicated to carefully defining the continual reinforcement learning problem. We formalize the notion of agents that "never stop learning" through a new mathematical language for analyzing and cataloging agents. Using this new language, we define a continual learning agent as one that can be understood as carrying out an implicit search process indefinitely, and continual reinforcement learning as the setting in which the best agents are all continual learning agents. We provide two motivating examples, illustrating that traditional views of multi-task reinforcement learning and continual supervised learning are special cases of our definition. Collectively, these definitions and perspectives formalize many intuitive concepts at the heart of learning, and open new research pathways surrounding continual learning agents.
Overview
CoChat Summary
TL;DR
The paper proposes a formal definition of continual reinforcement learning (CRL) as RL problems in which all optimal agents never stop their implicit search over an agent basis. It introduces the operators *generates* and *reaches* to make this precise, and shows multi-task RL and continual supervised learning as special cases.
Key findings
- CRL is defined for an RL problem \((e,v,\Lambda)\) with basis \(\Lambda_B \subset \Lambda\) generating \(\Lambda\): it is CRL if every optimal agent \(\lambda^*\) never reaches \(\Lambda_B\) (Definition 4.2). A continual learning agent is one where \(\Lambda_B \vdash_e \{\lambda\}\) and \(\lambda \not\rightsquigarrow_e \Lambda_B\) (Definition 4.1).
- Theorem 3.1 shows that for any agent-environment pair \((\lambda,e)\), there are infinitely many bases \(\Lambda_B\) such that \(\lambda \notin \Lambda_B\) and \(\Lambda_B \vdash_e \{\lambda\}\); Remark 3.2 says every agent either sometimes or never reaches such a basis.
- In a switching-MDP example with \(n=10\) underlying MDPs, switch probability \(0.001\), and \(\epsilon=0.15\), continual Q-learning with constant step size \(\alpha=0.1\) outperformed convergent Q-learning that anneals its step size to zero, using average reward with 95% confidence intervals over 250 runs.
- Theorem 4.1 states no basis element is optimal in CRL and \(\Lambda\) is not minimal; Theorem 4.2 shows *generates* is transitive, not commutative, reflexive, and its decision problem is undecidable; Theorem 4.3 shows *reaches* is not transitive, not commutative, reflexive, and its decision problem is undecidable.
Methods
- Conceptual/theoretical paper: formalizes agent-environment interfaces, histories, agents as history-based policies, environments, reward, and performance; introduces agent bases, learning rules, and the *generates* and *reaches* operators.
- Main analysis consists of formal definitions and theorems, with proofs deferred to Appendix B and additional analysis in Appendix C.
- Two motivating examples: a switching-MDP Q-learning experiment (as above) and a continual supervised learning setting adapted from Mai et al., argued to be a special case of the CRL definition under sufficient distributional shift.
Limitations
- Paper-stated: the definition is basis-dependent, so an agent may be continual relative to one basis but not another; the authors argue the basis is not arbitrary in practice because resource constraints and design choices restrict it. They also defer much analysis and proofs to the appendix and note further exploration of CRL properties as future work.
- Paper-stated: they remain agnostic to how reward is aggregated and assume the reward hypothesis, so all relevant goals are captured by a deterministic reward function.
- Evident from the text: empirical support is limited to one simple switching-MDP Q-learning comparison; no broader benchmark or real-world validation is reported. Key decision problems for the operators are undecidable, and the examples are illustrative rather than exhaustive.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 9 Not valid · 11 Not checked · 70 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Theorem 3.1.For any agent-environment pair (𝜆, 𝑒), there exists infinitely many choices of a basis, Λ B , such that both (1) 𝜆 ∉ Λ B , and (2) Λ B ⊢ 𝑒 {𝜆}.”
§ A Definition of Continual Reinforcement Learning > Definition 3.2.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
““Sometimes reaches” is not commutative: there exists a pair (Λ 1 , Λ 2 ) and 𝑒 ∈ ℰ such that”
§ A Definition of Continual Reinforcement Learning > 5. The decision problem,
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generates is transitive: For any triple (Λ 1 , Λ 2 , Λ 3 ) and 𝑒 ∈ ℰ, if Λ 1 ⊢ 𝑒 Λ 2 and Λ 2 ⊢ 𝑒 Λ 3 , then Λ 1 ⊢ 𝑒 Λ 3 .”
§ A Definition of Continual Reinforcement Learning > 4.5 Properties of Generates and Reaches
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“point (5.) shows that deciding whether a given agent sometimes reaches a basis in an environment is undecidable.”
§ A Definition of Continual Reinforcement Learning > 5. The decision problem,
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Since both variants of Q-learning can be viewed as searching over Λ B , the annealing variant that stops its search will under-perform compared to the continual approach.”
§ A Definition of Continual Reinforcement Learning > 4.2 CRL Examples
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
50 more claims
Create a free account to see all 90 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 14
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“While continual learning does not necessarily emphasize agent-gathered data or sustained interaction with the environment, it aligns with enactive views in emphasizing ongoing adaptation under non-stationarity.”
From Toward Enactive Artificial Intelligence (2026) · cites this paper as Abel et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Definition of Continual Reinforcement Learning > 4.5 Properties of Generates and Reaches
“Λ 1 ⊢ 𝑒 Λ 2 , but ¬(Λ 2 ⊢ 𝑒 Λ 1 ).”
§ A Definition of Continual Reinforcement Learning > □
“Note that since 𝜆 𝑖 is in both sets, and Λ 1 is a singleton, we know that Λ 2 ⊢ 𝑒 Λ 1 in any environment.But, by Proposition C.6, we know that Λ 1 cannot generate Λ 2 .”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Definition of Continual Reinforcement Learning > 2.2 Realizable Histories
“𝑒(𝑜 𝑘+1 | ℎ 𝑘 , 𝑎 𝑘 )𝜆(𝑎 𝑘 | ℎ 𝑘 ) > 0”
§ A Definition of Continual Reinforcement Learning > 𝒜 Actions 𝒪
“𝑒(𝑜 𝑘 | ℎ 𝑘 , 𝑎 𝑘 )𝜆(𝑎 𝑘 | ℎ 𝑘 ) > 0”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Definition of Continual Reinforcement Learning > Proof of Theorem 3.1.
“If odd, choose 𝜆 1 B , and if even, choose 𝜆 2 B .”
§ A Definition of Continual Reinforcement Learning > Proof of Theorem 3.1.
“Then, this agent will choose a different action distribution on the odd length histories: 𝜆 1 B (ℎ 𝑖+1 ) ≠ 𝜆(ℎ 𝑖+1 ), for 𝑖 any even natural number.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Definition of Continual Reinforcement Learning > 2.1 Agents and Environments
“Definition 2.3.An environment with respect to interface (𝒜, 𝒪) is a function 𝑒 : ℋ × 𝒜 → Δ(𝒪).”
§ A Definition of Continual Reinforcement Learning > 𝒜 Actions 𝒪
“𝑒 Environment 𝑒 : ℋ × 𝒜 → Δ(𝒜)”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117
Catastrophic forgetting in connectionist networks
French, R. (1999) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(99)01294-2 · Cited by 2,311
Toward an architecture for never-ending language learning
Carlson, A., Betteridge, J., Kisiel, B., Settles, B., Hruschka, E., & Mitchell, T. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7519 · Cited by 2,000
Transfer Learning for Reinforcement Learning Domains: A Survey
Taylor, M. E., & Stone, P. (2009) · doi:10.5555/1577069.1755839 · Cited by 1,488
An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6211 · Cited by 1,089
Experience Replay for Continual Learning
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T. P., & Wayne, G. (2018) · arXiv · doi:10.48550/arxiv.1811.11682 · Cited by 529
Research works printed but not linked: 20
- David Abel, Yuu Jinnai, Yue Guo, George Konidaris, and Michael L. Littman. Policy and value transfer in lifelong reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2018.
- Peter L Bartlett. Learning with a slowly changing distribution. In Proceedings of the Annual Workshop on Computational Learning Theory, 1992.
- Emma Brunskill and Lihong Li. PAC-inspired option discovery in lifelong reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2014.
- Anthony R. Cassandra, Leslie Pack Kaelbling, and Michael L. Littman. Acting optimally in partially observable stochastic domains. In Proceedings of the AAAI Conference on Artificiall Intelligence, 1994.
- Milton Friedman. Essays in positive economics. University of Chicago press, 1953.
- Haotian Fu, Shangqun Yu, Michael Littman, and George Konidaris. Model-based lifelong reinforcement learning with Bayesian exploration. Advances in Neural Information Processing Systems, 2022.
- Marcus Hutter. Universal artificial intelligence: Sequential decisions based on algorithmic probability. Springer Science & Business Media, 2004.
- Tor Lattimore. Theory of general reinforcement learning. PhD thesis, The Australian National University, 2014.
- Jan Leike. Nonparametric general reinforcement learning. PhD thesis, The Australian National University, 2016.
- Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, Ian Osband, and Zheng Wen. Reinforcement learning, bit by bit. Foundations and Trends in Machine Learning, 16(6):733-865, 2023. ISSN 1935-8237.
- Sultan J Majeed. Abstractions of general reinforcement Learning. PhD thesis, The Australian National University, 2021.
- Claire Monteleoni and Tommi Jaakkola. Online learning of non-stationary sequences. Advances in Neural Information Processing Systems, 16, 2003.
- Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
- Mark B Ring. Continual learning in reinforcement environments. PhD thesis, The University of Texas at Austin, 1994.
- Mark B Ring. Toward a formal framework for continual learning. In NeurIPS Workshop on Inductive Transfer, 2005.
- Paul Ruvolo and Eric Eaton. ELLA: An efficient lifelong learning algorithm. In Proceedings of the International Conference on Machine Learning, 2013.
- Daniel L Silver. Machine lifelong learning: Challenges and benefits for artificial general intelligence. In Proceedings of the Conference on Artificial General Intelligence, 2011.
- Richard S Sutton. Introduction: The challenge of reinforcement learning. In Reinforcement Learning, pages 1-3. Springer, 1992.
- Richard S Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
- Sebastian Thrun. Is learning the n-th thing any easier than learning the first? Advances in Neural Information Processing Systems, 1995.
Cited by
A Survey of Progress on Cooperative Multi-agent Reinforcement Learning in Open EnvironmentPreprint
Yuan, L., Zhang, Z., Li, L., Guan, C., & Yu, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.01058 · Cited by 111
Ammar, S., Lau, C. P., & Shihada, B. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3414622 · Cited by 79
Continual Learning: Applications and the Road ForwardPreprint
Verwimp, E., Aljundi, R., Ben-David, S., Bethge, M., Cossu, A., Gepperth, A., Hayes, T. L., Hüllermeier, E., Kanan, C., Kudithipudi, D., et al. (2023) · Transactions on machine learning research · doi:10.48550/arxiv.2311.11908 · Cited by 71
Continual Learning for Smart City: A Survey
Yang, L., Luo, Z., Zhang, S., Teng, F., & Li, T. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3447123 · Cited by 57
Continual Learning as Computationally Constrained Reinforcement Learning
Kumar, S., Marklund, H., Rao, A., Zhu, Y., Jeon, H. J., Liu, Y., & Roy, B. V. (2025) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000116 · Cited by 51
Self-citation
Metadata
- Authors
- Abel, D., Barreto, A., Roy, B. V., Precup, D., Hasselt, H. V., & Singh, S.
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 50377-50407
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133245600
- Cited by
- 36
- References
- 10
- Citations by year
- 2026: 15 · 2025: 7 · 2024: 13 · 2023: 1
- FWCI
- 6.45
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- continual reinforcement learning, definitions, lifelong learning, online learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.