A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Reinforcement learning (RL) algorithms find applications in inventory control, recommender systems, vehicular traffic management, cloud computing, and robotics. The real-world complications arising in these domains makes them difficult to solve with the basic assumptions underlying classical RL algorithms. RL agents in these applications often need to react and adapt to changing operating conditions. A significant part of research on single-agent RL techniques focuses on developing algorithms when the underlying assumption of stationary environment model is relaxed. This article provides a survey of RL methods developed for handling dynamically varying environment models. The goal of methods not limited by the stationarity assumption is to help autonomous agents adapt to varying operating conditions. This is possible either by minimizing the rewards lost during learning by RL agent or by finding a suitable policy for the RL agent that leads to efficient operation of the underlying system. A representative collection of these algorithms is discussed in detail in this work along with their categorization and their relative merits and demerits. Additionally, we also review works that are tailored to application domains. Finally, we discuss future enhancements for this field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512 · cited 2×
“A general model-agnostic meta-learning algorithm is proposed in [40].”
ResultMarkov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
PAGE, E. S. (1954) · Biometrika · doi:10.1093/biomet/41.1-2.100 · Cited by 5,739
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117
A Comprehensive Survey of Multiagent Reinforcement Learning
Busoniu, L., Babuska, R., & De Schutter, B. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2007.913919 · Cited by 2,235 · cited 1×
“ent dynamics is the next step in this progression, eliminating the drawback that RL algorithms are applicable only in domains with low risk, for e.g., video games [46] and pricing [60]. Multi-agentRL [16] is concernedwith learning in prescence of multiple agents. It can be considered as an extension of single-agent RL…”
BackgroundThe Netflix Recommender System: Algorithms, Business Value, and Innovation
Gomez-Uribe, C. A., & Hunt, N. (2016) · ACM Transactions on Management Information Systems · doi:10.1145/2843948 · Cited by 1,418
An introduction to deep reinforcement learning
François-Lavet, V., Henderson, P., Islam, R., Bellemare, M. G., & Pineau, J. (2018) · doi:10.1561/9781680835397 · Cited by 544
Child: A First Step Towards Continual Learning
Ring, M. B. (1998) · Learning to Learn · doi:10.1007/978-1-4615-5529-2_11 · Cited by 84
Contextual Markov Decision Processes
Hallak, A., Di Castro, D., & Mannor, S. (2015) · arXiv · doi:10.48550/arxiv.1502.02259 · Cited by 65
Cited by
Deep Learning Models for Serendipity Recommendations: A Survey and New Perspectives
Fu, Z., Niu, X., & Maher, M. L. (2024) · ACM Computing Surveys · doi:10.1145/3605145 · Cited by 30
Continual Learning as Computationally Constrained Reinforcement Learning
Kumar, S., Marklund, H., Rao, A., Zhu, Y., Jeon, H. J., Liu, Y., & Van Roy, B. (2023) · arXiv · doi:10.48550/arxiv.2307.04345 · Cited by 4
Metadata
- Authors
- Padakandla, S.
- Venue
- ACM Computing Surveys
- Published
- 2021-07-13
- Volume
- 54
- Issue
- 6
- Pages
- 1-25
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3027406032
- Cited by
- 237
- References
- 12
- FWCI
- 11.12
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.