The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Fast gradient-descent methods for temporal-difference learning with linear function approximation
Abstract
Sutton, Szepesvári and Maei (2009) recently introduced the first temporal-difference learning algorithm compatible with both linear function approximation and off-policy training, and whose complexity scales only linearly in the size of the function approximator. Although their gradient temporal difference (GTD) algorithm converges reliably, it can be very slow compared to conventional linear TD (on on-policy problems where TD is convergent), calling into question its practical utility. In this paper we introduce two new related algorithms with better convergence rates. The first algorithm, GTD2, is derived and proved convergent just as GTD was, but uses a different objective function and converges significantly faster (but still not as fast as conventional TD). The second new algorithm, linear TD with gradient correction, or TDC, uses the same update rule as conventional TD except for an additional term which is initially zero. In our experiments on small test problems and in a Computer Go application with a million features, the learning rate of this algorithm was comparable to that of conventional TD. This algorithm appears to extend linear TD to off-policy learning with no penalty in performance while only doubling computational requirements.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Self-citationBetween MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119
An analysis of temporal-difference learning with function approximation
Tsitsiklis, J. N., & Van Roy, B. (1997) · IEEE Transactions on Automatic Control · doi:10.1109/9.580874 · Cited by 1,194
Residual Algorithms: Reinforcement Learning with Function Approximation
Baird, L. (1995) · Machine Learning Proceedings 1995 · doi:10.1016/b978-1-55860-377-6.50013-x · Cited by 918
Linear Least-Squares algorithms for temporal difference learning
Bradtke, S. J., & Barto, A. G. (1996) · Machine Learning · doi:10.1007/bf00114723 · Cited by 599
Convergent activation dynamics in continuous time networks
Hirsch, M. W. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90018-x · Cited by 587
The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning
Borkar, V. S., & Meyn, S. P. (2000) · SIAM Journal on Control and Optimization · doi:10.1137/s0363012997331639 · Cited by 499
Stochastic approximation with two time scales
Borkar, V. S. (1997) · Systems & Control Letters · doi:10.1016/s0167-6911(97)90015-3 · Cited by 389
Technical Update: Least-Squares Temporal Difference Learning
Boyan, J. A. (2002) · Machine Learning · doi:10.1023/a:1017936530646 · Cited by 294
The convergence of TD(?) for general ?
Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992701 · Cited by 275
Sutton, R. S., Maei, H. R., & Szepesvári, C. (2008) · Cited by 273
Self-citationOff-Policy Temporal Difference Learning with Function Approximation
Precup, D., Sutton, R. S., & Dasgupta, S. (2001) · Cited by 243
Self-citationAntos, A., Szepesvári, C., & Munos, R. (2008) · Machine Learning · doi:10.1007/s10994-007-5038-2 · Cited by 219
Self-citationReinforcement learning of local shape in the game of go
Silver, D., Sutton, R. S., & Müller, M. (2007) · UCL Discovery (University College London) · Cited by 122
Self-citationReinforcement Learning Through Gradient Descent
Baird, L. C. (2018) · doi:10.1184/r1/6560651 · Cited by 85
Temporal-difference methods and Markov models
Barnard, E. (1993) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/21.229449 · Cited by 54
Incremental least-squares temporal difference learning
Geramifard, A. (2007) · doi:10.7939/r3-fkn7-mk03 · Cited by 53
Feature Construction for Reinforcement Learning in Hearts
Sturtevant, N., & White, A. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-75538-8_11 · Cited by 51
Cited by
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms
Zhang, K., Yang, Z., & Başar, T. (2021) · Studies in Systems, Decision and Control · doi:10.1007/978-3-030-60990-0_12 · Cited by 1,264
Algorithms for Reinforcement Learning
Szepesvári, C. (2010) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01551-9 · Cited by 1,049
Self-citationReinforcement Learning and Dynamic Programming Using Function Approximators
Busoniu, L., Babuska, R., De Schutter, B., & Ernst, D. (2017) · CRC Press eBooks · doi:10.1201/9781439821091 · Cited by 919
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020) · arXiv · doi:10.48550/arxiv.2005.01643 · Cited by 824
Applications of deep learning and reinforcement learning to biological data
Mahmud, M., Kaiser, M. S., Hussain, A., & Vassanelli, S. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2790388 · Cited by 765
Metadata
- Authors
- Sutton, R. S., Maei, H. R., Precup, D., Bhatnagar, S., Silver, D., Szepesvári, C., & Wiewiora, E.
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 993-1000
- OpenAlex
- W2075268401
- Cited by
- 496
- References
- 19
- FWCI
- 18.10
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.