The DOI matches its Crossref record; 95 of the 95 references it cites were found in a registry or an index.
Recent Advances in Hierarchical Reinforcement Learning
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Self-citationStatecharts: a visual formalism for complex systems
Harel, D. (1987) · Science of Computer Programming · doi:10.1016/0167-6423(87)90035-9 · Cited by 6,751
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Puterman, M. L. (1994) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316887 · Cited by 6,024
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
Some Studies in Machine Learning Using the Game of Checkers
Samuel, A. L. (1959) · IBM Journal of Research and Development · doi:10.1147/rd.33.0210 · Cited by 4,443
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149
(2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_5086 · Cited by 3,515
Multiagent Systems : A Modern Approach to Distributed Artificial Intelligence
Weiß, G. (2000) · Medical Entomology and Zoology · Cited by 3,458
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119
Markov games as a framework for multi-agent reinforcement learning
Littman, M. L. (1994) · Machine Learning Proceedings 1994 · doi:10.1016/b978-1-55860-335-6.50027-1 · Cited by 2,614
Singular Perturbation Methods in Control: Analysis and Design
Kokotović, P., Khalil, H. K., & O'Reilly, J. (1999) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611971118 · Cited by 2,485
Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents
Tan, M. (1993) · Machine Learning Proceedings 1993 · doi:10.1016/b978-1-55860-307-3.50049-6 · Cited by 1,810
On-line Q-learning Using Connectionist Systems
Rummery, G. A. (1994) · Cited by 1,477
Toward a modern theory of adaptive networks: Expectation and prediction.
Sutton, R. S., & Barto, A. G. (1981) · Psychological Review · doi:10.1037/0033-295x.88.2.135 · Cited by 1,450 · cited 1×
“Other early RL research was explicitly motivated by animal behavior and its neural basis [45] [33] [34] [71].”
BackgroundSelf-citationSome studies in machine learning using the game of checkers
Samuel, A. L. (2000) · IBM Journal of Research and Development · doi:10.1147/rd.441.0206 · Cited by 1,439
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
Dietterich, T. G. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.639 · Cited by 1,435
Dynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,325
Transition network grammars for natural language analysis
Woods, W. A. (1970) · Communications of the ACM · doi:10.1145/355598.362773 · Cited by 1,323
A Unified Framework for Hybrid Control: Model and Optimal Control Theory
Branicky, M. S., Borkar, V. S., & Mitter, S. K. (1998) · IEEE Transactions on Automatic Control · doi:10.1109/9.654885 · Cited by 1,313
An analysis of temporal-difference learning with function approximation
Tsitsiklis, J. N., & Van Roy, B. (1997) · IEEE Transactions on Automatic Control · doi:10.1109/9.580874 · Cited by 1,194
The Complexity of Decentralized Control of Markov Decision Processes
Bernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002) · Mathematics of Operations Research · doi:10.1287/moor.27.4.819.297 · Cited by 1,192
Introduction to Stochastic Dynamic Programming.
Berry, D. A., & Ross, S. M. (1986) · Journal of the American Statistical Association · doi:10.2307/2289262 · Cited by 1,133
Learning to act using real-time dynamic programming
Barto, A. G., Bradtke, S. J., & Singh, S. P. (1995) · Artificial Intelligence · doi:10.1016/0004-3702(94)00011-o · Cited by 1,104
Self-citationGeneralization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding
Sutton, R. S. (1995) · Cited by 1,072
Learning and executing generalized robot plans
Fikes, R. E., Hart, P. E., & Nilsson, N. J. (1972) · Artificial Intelligence · doi:10.1016/0004-3702(72)90051-3 · Cited by 1,019
A model for reasoning about persistence and causation
Dean, T., & Kanazawa, K. (1989) · Computational Intelligence · doi:10.1111/j.1467-8640.1989.tb00324.x · Cited by 999
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Medical Entomology and Zoology · Cited by 971
The hierarchical hidden Markov model: Analysis and applications
Fine, S., Singer, Y., & Tishby, N. (1998) · Machine Learning · doi:10.1023/a:1007469218079 · Cited by 906
TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play
Tesauro, G. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.215 · Cited by 787
Practical issues in temporal difference learning
Tesauro, G. (1992) · Machine Learning · doi:10.1007/bf00992697 · Cited by 787
On the Convergence of Stochastic Iterative Dynamic Programming Algorithms
Jaakkola, T., Jordan, M. I., & Singh, S. P. (1994) · Neural Computation · doi:10.1162/neco.1994.6.6.1185 · Cited by 770
Some Studies in Machine Learning Using the Game of Checkers. II—Recent Progress
Samuel, A. L. (1967) · IBM Journal of Research and Development · doi:10.1147/rd.116.0601 · Cited by 695
Reinforcement Learning with Hierarchies of Machines
Parr, R., & Russell, S. (1997) · Cited by 618
Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms
Singh, S., Jaakkola, T., Littman, M. L., & Szepesvári, C. (2000) · Machine Learning · doi:10.1023/a:1007678930559 · Cited by 609
Tractable Inference for Complex Stochastic Processes
Boyen, X., & Koller, D. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.7362 · Cited by 471
Automatic Discovery of Subgoals in Reinforcement Learning using Diverse Density
McGovern, A., & Barto, A. G. (2001) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 418
Self-citationReinforcement learning with selective perception and hidden state
McCallum, A. K., & Ballard, D. H. (1996) · Cited by 411
Average reward reinforcement learning: Foundations, algorithms, and empirical results
Mahadevan, S. (1996) · Machine Learning · doi:10.1007/bf00114727 · Cited by 385
Self-citationAchieving Artificial Intelligence through Building Robots
Brooks, R. A. (1986) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 329
A Reinforcement Learning Method for Maximizing Undiscounted Rewards
Schwartz, A. (1993) · Machine Learning Proceedings 1993 · doi:10.1016/b978-1-55860-307-3.50045-9 · Cited by 308
Elevator Group Control Using Multiple Reinforcement Learning Agents
Crites, R. H., & Barto, A. G. (1998) · Machine Learning · doi:10.1023/a:1007518724497 · Cited by 275
Self-citationReinforcement Learning Methods for Continuous-Time Markov Decision Problems
Bradtke, S. J., & Duff, M. O. (1994) · Cited by 263
Werbos, P. (1987) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1987.289329 · Cited by 251
DERVISH An Office-Navigating Robot
Nourbakhsh, I., Powers, R., & Birchfield, S. T. (1995) · AI Magazine · doi:10.1609/aimag.v16i2.1133 · Cited by 244
Temporal abstraction in reinforcement learning
Precup, D., & Sutton, R. S. (2000) · Cited by 236
Morimoto, J., & Doya, K. (2005) · Neural Computation · doi:10.1162/0899766053011528 · Cited by 232
The Complexity of Decentralized Control of Markov Decision Processes
Bernstein, D. S., Shlomo Zilberstein, B., & Immerman, N. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3836 · Cited by 232
Hierarchical control and learning for markov decision processes
Parr, R., & Russell, S. (1998) · Cited by 221
Solving Semi-Markov Decision Problems Using Average Reward Reinforcement Learning
Das, T. K., Gosavi, A., Mahadevan, S., & Marchalleck, N. (1999) · Management Science · doi:10.1287/mnsc.45.4.560 · Cited by 218
Self-citationFinding Structure in Reinforcement Learning
Thrun, S., & Schwartz, A. (1994) · neural information processing systems · Cited by 218
Reinforcement Learning for Dynamic Channel Allocation in Cellular Telephone Systems
Singh, S. P., & Bertsekas, D. P. (1996) · Cited by 214
Xavier: a robot navigation architecture based on partially observable Markov decision process models
Koenig, S., & Simmons, R. G. (1998) · The MIT Press eBooks · Cited by 189
Discovering Hierarchy in Reinforcement Learning with HEXQ
Hengst, B. (2002) · Cited by 181
Singular Perturbation Methodology in Control Systems
Naidu, D. S. (1988) · Institution of Engineering and Technology eBooks · doi:10.1049/pbce034e · Cited by 176
Learning topological maps with weak local odometric information
Shatkay, H., & Kaelbling, L. P. (1997) · Cited by 164
Scaling Reinforcement Learning toward RoboCup Soccer
Stone, P., & Sutton, R. S. (2001) · Cited by 164
Hierarchical multi-agent reinforcement learning
Makar, R., Mahadevan, S., & Ghavamzadeh, M. (2001) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/375735.376302 · Cited by 155
Self-citationThe Hedonistic Neuron: A Theory of Memory, Learning and Intelligence
Edwards, J. S., & Klopf, A. H. (1983) · Journal of the Operational Research Society · doi:10.1057/jors.1983.154 · Cited by 155
A Heuristic Approach to the Discovery of Macro-Operators
Iba, G. A. (1989) · Machine Learning · doi:10.1023/a:1022693717366 · Cited by 152
Lyapunov design for safe reinforcement learning
Perkins, T. J., & Barto, A. G. (2003) · doi:10.5555/944919.944955 · Cited by 148
Self-citationSingular Perturbation Methodology in Control Systems
Naidu (1988) · Cited by 135
Theory of neural-analog reinforcement systems and its application to the brain-model problem
Minsky, M. (1954) · Cited by 130
Learning to solve problems by searching for macro-operators
Korf, R. E. (1983) · Pitman Advanced Pub. Program eBooks · Cited by 119
Reinforcement learning with a hierarchy of abstract models
Singh, S. P. (1992) · Cited by 108
Multi-time Models for Temporally Abstract Planning
Precup, D., & Sutton, R. S. (1997) · Cited by 108
Programmable Reinforcement Learning Agents
André, D., & Russell, S. (2000) · Cited by 106
Theoretical results on reinforcement learning with temporally abstract options
Precup, D., Sutton, R. S., & Singh, S. P. (1998) · Lecture notes in computer science · doi:10.1007/bfb0026709 · Cited by 103
Introduction to stochastic dynamic programming
Ross, S. M., Birnbaum, Z. W., & Lukacs, E. (1983) · Cited by 97
Learning Hierarchical Control Structures for Multiple Tasks and Changing Environments
Digney, B. L. (1998) · The MIT Press eBooks · doi:10.7551/mitpress/3119.003.0050 · Cited by 95
Autonomous discovery of temporal abstractions from interaction with an environment
Mcgovern, E. A., & Barto, A. G. (2002) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 78
Self-citationA time aggregation approach to Markov decision processes
Cao, X., Ren, Z., Bhatnagar, S., Fu, M. C., & Marcus, S. I. (2002) · Automatica · doi:10.1016/s0005-1098(01)00282-5 · Cited by 73
Self-Improving Factory Simulation using Continuous-time Average-Reward Reinforcement Learning
Mahadevan, S., Das, T. K., & Gosavi, A. (2007) · Cited by 67
Self-citationA feedback control structure for on-line learning tasks
Huber, M., & Grupen, R. A. (1997) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(97)00044-4 · Cited by 64
Multilayer control of large Markov chains
Forestier, J.-P., & Varaiya, P. P. (1978) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1978.1101707 · Cited by 64
Learning to Improve Coordinated Actions in Cooperative Distributed Problem-Solving Environments
Sugawara, T., & Lesser, V. (1998) · Machine Learning · doi:10.1023/a:1007510522680 · Cited by 59
Theocharous, G., & Mahadevan, S. (2003) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2002.1014730 · Cited by 49
Self-citationLearning to Take Concurrent Actions
Rohanimanesh, K., & Mahadevan, S. (2002) · Cited by 44
Self-citationAutomated State Abstraction for Options using the U-Tree Algorithm
Jönsson, A., & Barto, A. G. (2000) · Cited by 43
Self-citationKeepaway Soccer: A Machine Learning Test bed
Stone, P., & Sutton, R. S. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45603-1_22 · Cited by 41
Practical Issues in Temporal Difference Learning
Tesauro, G. J. (1992) · Machine Learning · doi:10.1023/a:1022624705476 · Cited by 41
Yin, G. G., & Zhang, Q. (2012) · Stochastic modelling and applied probability · doi:10.1007/978-1-4614-4346-9_7 · Cited by 38
Integrating Experimentation and Guidance in Relational Reinforcement Learning
Driessens, K., & Džeroski, S. (2002) · Lirias · Cited by 34
Decision-Theoretic Planning with Concurrent Temporally Extended ActionsPreprint
Rohanimanesh, K., & Mahadevan, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.2307 · Cited by 34
Self-citationHierarchical learning and planning in partially observable markov decision processes
Mahadevan, S., & Theocharous, G. (2002) · Cited by 30
Self-citationContinuous-Time Hierarchical Reinforcement Learning
Ghavamzadeh, M., & Mahadevan, S. (2001) · Cited by 27
Self-citationScaling Reinforcement Learning Algorithms by Learning Variable Temporal Resolution Models
Singh, S. P. (1992) · Elsevier eBooks · doi:10.1016/b978-1-55860-247-2.50058-9 · Cited by 27
Average Reward Reinforcement Learning: Foundations, Algorithms, and Empirical Results
Mahadevan, S. (1996) · Machine Learning · doi:10.1023/a:1018064306595 · Cited by 26
Self-citationLarge-scale dynamic optimization using teams of reinforcement learning agents
Crites, R. H., & Barto, A. G. (1996) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 25
Self-citationDigney, B. L. (1996) · The MIT Press eBooks · doi:10.7551/mitpress/3118.003.0044 · Cited by 23
Autonomous Discovery of Abstractions through Interaction with an Environment
McGovern, A. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45622-8_34 · Cited by 22
Lyapunov-Constrained Action Sets for Reinforcement Learning
Perkins, T. J., & Barto, A. G. (2001) · International Conference on Machine Learning · Cited by 20
Self-citationHierarchically Optimal Average Reward Reinforcement Learning
Ghavamzadeh, M., & Mahadevan, S. (2002) · Cited by 14
Self-citation
Cited by
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
Deep reinforcement learning for autonomous driving: A survey
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A. A. A., Yogamani, S., & Perez, P. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3054625 · Cited by 2,342
Brochu, E., Cora, V. M., & de Freitas, N. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1012.2599 · Cited by 2,011
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 770
Metadata
- Authors
- Barto, A. G., & Mahadevan, S.
- Venue
- Discrete Event Dynamic Systems
- Published
- 2003-01
- Volume
- 13
- Issue
- 1-2
- Pages
- 41-77
- ISSN
- 0924-6703 (ISSN-L)
- 1573-7594
- OpenAlex
- W1592847719
- Cited by
- 1,024
- References
- 95
- FWCI
- 34.28
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.