PAC model-free reinforcement learning
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
For a Markov Decision Process with finite state (size S) and action spaces (size A per state), we propose a new algorithm---Delayed Q-Learning. We prove it is PAC, achieving near optimal performance except for Õ(SA) timesteps using O(SA) space, improving on the Õ(S2 A) bounds of best previous algorithms. This result proves efficient reinforcement learning is possible without learning a model of the MDP from experience. Learning takes place from a single continuous thread of experience---no resets nor parallel sampling is used. Beyond its smaller storage and experience requirements, Delayed Q-learning's per-experience computation cost is much less than that of previous PAC algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Learning to act using real-time dynamic programming
Barto, A. G., Bradtke, S. J., & Singh, S. P. (1995) · Artificial Intelligence · doi:10.1016/0004-3702(94)00011-o · Cited by 1,104
Near-Optimal Reinforcement Learning in Polynomial Time
Kearns, M., & Singh, S. (2002) · Machine Learning · doi:10.1023/a:1017984413808 · Cited by 792
On the Sample Complexity of Reinforcement Learning
Kakade, S. M. (2003) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 676
Even-Dar, E., & Mansour, Y. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_39 · Cited by 314
Finite-Sample Convergence Rates for Q-Learning and Indirect Algorithms
Kearns, M. J., & Singh, S. P. (1998) · Cited by 200
A theoretical analysis of Model-Based Interval Estimation
Strehl, A. L., & Littman, M. L. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102459 · Cited by 135
Self-citationEfficient reinforcement learning
Fiechter, C.-N. (1994) · Proceedings of the seventh annual conference on Computational learning theory - COLT '94 · doi:10.1145/180139.181019 · Cited by 72
Cited by
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733
A Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
(2015) · Journal of Machine Learning Research · Cited by 1,119
Algorithms for Reinforcement Learning
Szepesvári, C. (2010) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01551-9 · Cited by 1,049
Reinforcement learning for demand response: A review of algorithms and modeling techniques
Vázquez-Canteli, J. R., & Nagy, Z. (2019) · Applied Energy · doi:10.1016/j.apenergy.2018.11.002 · Cited by 883
Reinforcement learning algorithms: A brief survey
Shakya, A. K., Pillai, G., & Chakrabarty, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120495 · Cited by 656
Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms
Li, L., Chu, W., Langford, J., & Wang, X. (2011) · Proceedings of the fourth ACM international conference on Web search and data mining · doi:10.1145/1935826.1935878 · Cited by 443
Self-citationAn analysis of model-based Interval Estimation for Markov Decision Processes
Strehl, A. L., & Littman, M. L. (2008) · Journal of Computer and System Sciences · doi:10.1016/j.jcss.2007.08.009 · Cited by 428
“In addition, Strehl et al. (2006b) provide a modelfree algorithm called Delayed Q-learning that is PAC-MDP and has a resemblance to Q-learning.”
BackgroundMethodSelf-citationAliramezani, M., Koch, C. R., & Shahbakhti, M. (2022) · Progress in Energy and Combustion Science · doi:10.1016/j.pecs.2021.100967 · Cited by 288
An object-oriented representation for efficient reinforcement learning
Diuk, C., Cohen, A., & Littman, M. L. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390187 · Cited by 226
Self-citationLearning Combinatorial Optimization on Graphs: A Survey With Applications to Networking
Vesselinova, N., Steinert, R., Perez-Ramirez, D. F., & Boman, M. (2020) · IEEE Access · doi:10.1109/access.2020.3004964 · Cited by 164
“[63] clarify that the evaluation of reinforcement algorithms involves in addition to the space-”
BackgroundHierarchical Reinforcement Learning: A Survey and Open Research Challenges
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4010009 · Cited by 129
Multi-Agent Reinforcement Learning for Traffic Signal Control: A Cooperative Approach
Kolat, M., Kővári, B., Bécsi, T., & Aradi, S. (2023) · Sustainability · doi:10.3390/su15043479 · Cited by 94
“Although reinforcement learning has reached success in the past [18–21], previous approaches faced several limitations such as dimensional, memory, computational or sample complexity issues [22].”
BackgroundA Survey of Exploration Methods in Reinforcement Learning
Amin, S., Gomrokchi, M., Satija, H., van Hoof, H., & Precup, D. (2021) · arXiv · doi:10.48550/arxiv.2109.00157 · Cited by 41
Count-based exploration with the successor representation
Machado, M. C., Bellemare, M. G., & Bowling, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5955 · Cited by 19
PAC Reinforcement Learning with Rich Observations
Krishnamurthy, A., Agarwal, A., & Langford, J. (2016) · arXiv · doi:10.48550/arxiv.1602.02722 · Cited by 13
Self-citation
Metadata
- Authors
- Strehl, A. L., Li, L., Wiewiora, E., Langford, J., & Littman, M. L.
- Venue
- Proceedings of the 23rd international conference on Machine learning - ICML '06
- Publisher
- ACM Press
- Published
- 2006
- Pages
- 881-888
- OpenAlex
- W2129670787
- Cited by
- 384
- References
- 10
- FWCI
- 5.28
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.