The DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index.
Recent Advances in Hierarchical Reinforcement Learning
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Hierarchical control has been suggested as a general framework for the control of complex systems (Barto and Mahadevan, 2003), and the octopus is an exemplar of this form of control (Zullo et al., 2009; Wayne and Abbott, 2014).”
From Lessons for Robotics From the Control Architecture of the Octopus (2022) · cites this paper as Barto and Mahadevan, 2003
Citations
Cites
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Statecharts: a visual formalism for complex systems
Harel, D. (1987) · Science of Computer Programming · doi:10.1016/0167-6423(87)90035-9 · Cited by 6,771
Puterman, M. L. (1994) · Wiley series in probability and statistics · doi:10.1002/9780470316887 · Cited by 6,057
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149
Multiagent Systems : A Modern Approach to Distributed Artificial Intelligence
Weiß, G. (2000) · Medical Entomology and Zoology · Cited by 3,458
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119
Singular Perturbation Methods in Control: Analysis and Design
Kokotović, P. V., Khalil, H. K., & O’Reilly, J. (1999) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611971118 · Cited by 2,504
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
Dietterich, T. G. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.639 · Cited by 1,526
Toward a modern theory of adaptive networks: Expectation and prediction.
Sutton, R. S., & Barto, A. G. (1981) · Psychological Review · doi:10.1037/0033-295x.88.2.135 · Cited by 1,450
Self-citationDynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,348
Transition network grammars for natural language analysis
Woods, W. A. (1970) · Communications of the ACM · doi:10.1145/355598.362773 · Cited by 1,334
A Unified Framework for Hybrid Control: Model and Optimal Control Theory
Branicky, M. S., Borkar, V. S., & Mitter, S. K. (1998) · IEEE Transactions on Automatic Control · doi:10.1109/9.654885 · Cited by 1,313
An analysis of temporal-difference learning with function approximation
Tsitsiklis, J. N., & Roy, B. V. (1997) · IEEE Transactions on Automatic Control · doi:10.1109/9.580874 · Cited by 1,290
The Complexity of Decentralized Control of Markov Decision Processes
Bernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002) · Mathematics of Operations Research · doi:10.1287/moor.27.4.819.297 · Cited by 1,192
Learning to act using real-time dynamic programming
Barto, A. G., Bradtke, S. J., & Singh, S. P. (1995) · Artificial Intelligence · doi:10.1016/0004-3702(94)00011-o · Cited by 1,129
Self-citationLearning and executing generalized robot plans
Fikes, R. E., Hart, P. E., & Nilsson, N. J. (1972) · Artificial Intelligence · doi:10.1016/0004-3702(72)90051-3 · Cited by 1,025
A model for reasoning about persistence and causation
Dean, T., & Kanazawa, K. (1989) · Computational Intelligence · doi:10.1111/j.1467-8640.1989.tb00324.x · Cited by 999
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Medical Entomology and Zoology · Cited by 971
The hierarchical hidden Markov model: Analysis and applications
Fine, S., Singer, Y., & Tishby, N. (1998) · Machine Learning · doi:10.1023/a:1007469218079 · Cited by 906
TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play
Tesauro, G. J. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.215 · Cited by 821
On the Convergence of Stochastic Iterative Dynamic Programming Algorithms
Jaakkola, T., Jordan, M. I., & Singh, S. P. (1994) · Neural Computation · doi:10.1162/neco.1994.6.6.1185 · Cited by 801
Some Studies in Machine Learning Using the Game of Checkers. II—Recent Progress
Samuel, A. L. (1967) · IBM Journal of Research and Development · doi:10.1147/rd.116.0601 · Cited by 700
Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms
Singh, S. P., Jaakkola, T., Littman, M. L., & Szepesvári, C. (2000) · Machine Learning · doi:10.1023/a:1007678930559 · Cited by 637
Reinforcement Learning with Hierarchies of Machines
Parr, R., & Russell, S. (1997) · Cited by 618
Tractable Inference for Complex Stochastic ProcessesPreprint
Boyen, X., & Koller, D. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.7362 · Cited by 477
Automatic Discovery of Subgoals in Reinforcement Learning using Diverse Density
McGovern, A., & Barto, A. G. (2001) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 416
Self-citationA Reinforcement Learning Method for Maximizing Undiscounted Rewards
Schwartz, A. (1993) · Elsevier eBooks · doi:10.1016/b978-1-55860-307-3.50045-9 · Cited by 310
Achieving Artificial Intelligence through Building Robots
Brooks, R. A. (1986) · Defense Technical Information Center (DTIC) · Cited by 304
Elevator Group Control Using Multiple Reinforcement Learning Agents
Crites, R. H., & Barto, A. G. (1998) · Machine Learning · doi:10.1023/a:1007518724497 · Cited by 275
Self-citationWerbos, P. J. (1987) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1987.289329 · Cited by 249
Solving Semi-Markov Decision Problems Using Average Reward Reinforcement Learning
Das, T. K., Gosavi, A., Mahadevan, S., & Marchalleck, N. (1999) · Management Science · doi:10.1287/mnsc.45.4.560 · Cited by 223
Self-citationHierarchical control and learning for markov decision processes
Parr, R., & Russell, S. (1998) · Cited by 221
Discovering Hierarchy in Reinforcement Learning with HEXQ
Hengst, B. (2002) · Cited by 184
Finding Structure in Reinforcement Learning
Thrun, S., & Schwartz, A. (1994) · Cited by 176
Singular Perturbation Methodology in Control Systems
Naidu, D. S. (1988) · Institution of Engineering and Technology eBooks · doi:10.1049/pbce034e · Cited by 176
Scaling Reinforcement Learning toward RoboCup Soccer
Stone, P., & Sutton, R. S. (2001) · Cited by 166
Hierarchical multi-agent reinforcement learning
Makar, R., Mahadevan, S., & Ghavamzadeh, M. (2001) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/375735.376302 · Cited by 155
Self-citationLyapunov design for safe reinforcement learning
Perkins, T. J., & Barto, A. G. (2003) · doi:10.5555/944919.944955 · Cited by 148
Self-citationXavier: a robot navigation architecture based on partially observable Markov decision process models
Koenig, S., & Simmons, R. G. (1998) · Cited by 147
A Heuristic Approach to the Discovery of Macro-Operators
Iba, G. A. (1989) · Machine Learning · doi:10.1023/a:1022693717366 · Cited by 143
The Hedonistic Neuron: A Theory of Memory, Learning and Intelligence
Edwards, J. S. (1983) · Journal of the Operational Research Society · doi:10.1057/jors.1983.154 · Cited by 137
Theory of neural-analog reinforcement systems and its application to the brain-model problem
Minsky, M. (1954) · Cited by 130
Learning to solve problems by searching for macro-operators
Korf, R. E. (1983) · Pitman Advanced Pub. Program eBooks · Cited by 119
Reinforcement learning with a hierarchy of abstract models
Singh, S. P. (1992) · Cited by 108
Programmable Reinforcement Learning Agents
André, D., & Russell, S. (2000) · Cited by 106
Theoretical results on reinforcement learning with temporally abstract options
Precup, D., Sutton, R. S., & Singh, S. P. (1998) · Lecture notes in computer science · doi:10.1007/bfb0026709 · Cited by 103
Advances in Neural Information Processing Systems 17: Proceedings of the 2004 Conference
Lund, R. (2006) · Quarterly Publications of the American Statistical Association · Cited by 78
Brain Function and Adaptive Systems: A Heterostatic Theory
Klopf, A. H. (1972) · Defense Technical Information Center (DTIC) · Cited by 78
A time aggregation approach to Markov decision processes
Cao, X., Ren, Z., Bhatnagar, S., Fu, M. C., & Marcus, S. I. (2002) · Automatica · doi:10.1016/s0005-1098(01)00282-5 · Cited by 73
Self-Improving Factory Simulation using Continuous-time Average-Reward Reinforcement Learning
Mahadevan, S., Das, T. K., & Gosavi, A. (2007) · Cited by 67
Self-citationA feedback control structure for on-line learning tasks
Huber, M., & Grupen, R. A. (1997) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(97)00044-4 · Cited by 64
Multilayer control of large Markov chains
Forestier, J.-P., & Varaiya, P. P. (1978) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1978.1101707 · Cited by 64
Learning to Improve Coordinated Actions in Cooperative Distributed Problem-Solving Environments
Sugawara, T., & Lesser, V. (1998) · Machine Learning · doi:10.1023/a:1007510522680 · Cited by 59
Practical Issues in Temporal Difference Learning
Tesauro, G. J. (1992) · Machine Learning · doi:10.1023/a:1022624705476 · Cited by 41
Decision-Theoretic Planning with Concurrent Temporally Extended ActionsPreprint
Rohanimanesh, K., & Mahadevan, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.2307 · Cited by 34
Self-citationIntegrating Experimentation and Guidance in Relational Reinforcement Learning
Driessens, K., & Džeroski, S. (2002) · Lirias · Cited by 34
Continuous-Time Hierarchical Reinforcement Learning
Ghavamzadeh, M., & Mahadevan, S. (2001) · Cited by 27
Self-citationScaling Reinforcement Learning Algorithms by Learning Variable Temporal Resolution Models
Singh, S. P. (1992) · Elsevier eBooks · doi:10.1016/b978-1-55860-247-2.50058-9 · Cited by 27
Average Reward Reinforcement Learning: Foundations, Algorithms, and Empirical Results
Mahadevan, S. (1996) · Machine Learning · doi:10.1023/a:1018064306595 · Cited by 26
Self-citationDigney, B. L. (1996) · The MIT Press eBooks · doi:10.7551/mitpress/3118.003.0044 · Cited by 23
Autonomous Discovery of Abstractions through Interaction with an Environment
McGovern, A. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45622-8_34 · Cited by 22
Lyapunov-Constrained Action Sets for Reinforcement Learning
Perkins, T. J., & Barto, A. G. (2001) · Cited by 19
Self-citationHierarchically Optimal Average Reward Reinforcement Learning
Ghavamzadeh, M., & Mahadevan, S. (2002) · Cited by 14
Self-citation
Research works printed but not linked: 27
- Bertsekas, D. P., and Tsitsiklis, J. N. 1996. Neuro-Dynamic Programming. Belmont, MA: Athena Scientific.
- Bradtke, S. J., and Duff. M. O. 1995. Reinforcement learning methods for continuous-time markov decision problems. In G. Tesauro. D. S. Touretzky and T. Leen (eds.). Advances in Neural Information Processing Systems: Proceedings of the 1994 Conference. Cambridge. MA: MIT Press. pp. 393-400.
- Crites. R. H. 1996. Large-Scale Dvnaniic Optimization Using Teams of Reinforcement Learning Agents. Ph.D. thesis. Amherst. MA: University of Massachusetts.
- Digney, B. 1998. Learning hierarchical control structure from multiple tasks and changing environments. In From Animals to Animals 5: The Fifth Conference on Simulation of Adaptive Behavior. Carnbndge, MA: MIT Press.
- Grudic. G. Z., and Ungar. L. H. 2000. Localizing search in reinforcement learning. In Proceedings of the 18th National Conference on Artificial Intelligence, (AAAI-00), pp. 590-595.
- Hernandez, N., and Mahadevan. S. 2001. Hierarchical memory-based reinforcement learning. In Advances in Neural information Processing Systems: Proceedings of the 2000 conference. Cambridge, MA: MIT Press, pp. 1047-1053.
- Howard, R. A. 1971. Dynamic Probabilistic Systems: Semi-Markar and Decision Processes. New York: Wiley.
- Jonsson. A., and Barto. A. G. 2001. Automated state abstraction for options using the li-tree algorithm. In Advances in Neural Information Processing Systems: Proceedings of the 2000 Conference. Cambridge, MA: MIT Press, pp. 1054-1060.
- Littman. M. 1994. Markov games as a framework for multi-agent reinforcement learning. In Proceedings of the Eleventh International Conference on Machine Learning. San Francisco, CA: Morgan Kaufmann, pp. 157-163.
- McCalIum, A. K. 1996. Reinforcement Learning with Selective Perceptioin and Hidden State. Ph.D. thesis, University of Rochester.
- Nourbakhsh, I., Powers, R., and Birchfield, S. 1995. Dervish: An office-navigation robot. Al Magazine 16(2): 53-60.
- Precup, D. 2000. Temporal Abstraction in Reinforcentent Learning. Ph.D. thesis, Amherst, MA: University of Massachusetts.
- Precup. D., and Sutton, R. S. 1998. Multi-time models for temporally abstract planning. In Advances in Neural information Processing Systems: Proceedings of the 1997 Conference. Cambridge MA: MIT Press, pp. 1050-1056.
- Ross. S. 1983. Introduction to Stochastic Dynamic Programming. New York: Academic Press.
- Rummery, G. A., and Niranjan, M. 1994. On-line q-learrnng using connectionist systems. Technical Report CUED/F-INFENG/'FR 166, Cambridge University Engineering Department.
- Samuel. A. L. 1963. Some studies in machine learning using the game of checkers. IBM Journal on Research and Developntent 3: 211-229, 1959. Reprinted in E. A. Feigenbaurn and J. Feldman (eds.), Computers and Thought. New York: McGraw-Hill, pp. pp71-105.
- Shatkay, H., and Kaelbling. L. P. 1997. Learning topological maps with weak local odometric information. In IJC4I (2). 920-929.
- Singh. S., and Bertsekas, D. 1997. Reinforcement learning for dynamic channel allocation in cellular telephone systems. In Advances in Neural Information Processing Systems: Proceedings of the 1996 conference. Cambridge, MA: MIT Press, pp. 974-980.
- Sutton, R. S. 1996. Generalization in reinforcement leaming: Successful examples using sparse coarse coding. In D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo (eds.). Advances in Neural Information Processing Svstenis: Proceedings of the 1995 Conference. Cambridge. MA: MIT Press. pp. 1038-1044.
- Sutton. R. S., and Barto, A. G. 1998. Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
- Tan, M. 1993. Multi-agent reinforcement leaming: Independent vs. cooperative agents. In Proceedings of the Tenth International Conference on Mac/tine Learning. San Francisco, CA: Morgan Kaufmann, pp. 330-337.
- Theocharous, G. 2002. Hierarchical Learning and Planning in Partially Observable Markov Derision Processes. Ph.D. Thesis, Michigan State University.
- Theocharous, G., and Mahadevan. S. 2002. Approximate planning with hierarchical partially observable markov decision processs for robot navigation. In Proceedings of the IEEE International Conference on Robotics and Autontation (ICRA).
- Theocharous, G., Rohanimanesh. K., and Mahadevan, S. 2001. Learning hierarchical partially observable markov decision processs for robot navigation. In Proceedings of the IEEE International Conference o,i Robotics and Automation (ICRA).
- Watkins, C. J. C. H. 1989. Learning front Delayed Rewards. Ph.D. thesis, Cambridge, England: Cambridge University.
- Watkins, C. J. C. H., and Dayan, P. 1992. Q-learning. Machine Learning 8: 279-292.
- Werbos, P. J. 1977. Advanced forecasting methods for global crisis warning and models of intelligence. General Systems Yearbook 22: 25-38.
Cited by
Sorted by citations
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
Deep Reinforcement Learning: An Overview
Mousavi, S. S., Schukat, M., & Howley, E. (2017) · Lecture notes in networks and systems · doi:10.1007/978-3-319-56991-8_32 · Cited by 1,045
Reinforcement learning: The Good, The Bad and The Ugly
Dayan, P., & Niv, Y. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.08.003 · Cited by 733
Hierarchical models of behavior and prefrontal function
Botvinick, M. M. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.02.009 · Cited by 532
Metadata
- Authors
- Barto, A. G., & Mahadevan, S.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Discrete Event Dynamic Systems
- Published
- 2003-10
- Volume
- 13
- Issue
- 4
- Pages
- 341-379
- ISSN
- 0924-6703 (ISSN-L)
- 1573-7594
- OpenAlex
- W4249441547
- Cited by
- 317
- References
- 64
- Citations by year
- 2026: 29 · 2025: 44 · 2024: 8 · 2023: 4 · 2022: 10 · 2021: 14 · 2020: 13 · 2019: 20 · 2018: 21 · 2017: 17 · 2016: 11 · 2015: 12 · 2014: 18 · 2013: 20 · 2012: 23
- FWCI
- 2.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Discrete Event Dynamic Systems (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Keywords
- hierarchical reinforcement learning, temporal abstraction, semi-Markov decision process, curse of dimensionality, hierarchical control architecture, concurrent actions, multi-agent coordination, hierarchical memory, partial observability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
