The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Hierarchical Reinforcement Learning: A Comprehensive Survey
Abstract
Hierarchical Reinforcement Learning (HRL) enables autonomous decomposition of challenging long-horizon decision-making tasks into simpler subtasks. During the past years, the landscape of HRL research has grown profoundly, resulting in copious approaches. A comprehensive overview of this vast landscape is necessary to study HRL in an organized manner. We provide a survey of the diverse HRL approaches concerning the challenges of learning hierarchical policies, subtask discovery, transfer learning, and multi-agent learning using HRL. The survey is presented according to a novel taxonomy of the approaches. Based on the survey, a set of important open problems is proposed to motivate the future research in HRL. Furthermore, we outline a few suitable task domains for evaluating the HRL approaches and a few interesting examples of the practical applications of HRL in the Supplementary Material.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Artificial intelligence: A modern approach
Nilsson, N. J. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(96)00007-0 · Cited by 10,583
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Advances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 8,890
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099
Proceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,680
Baum, L. E., Petrie, T., Soules, G., & Weiss, N. (1970) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177697196 · Cited by 4,297
Encyclopedia of Machine Learning
(2010) · Springer eBooks · doi:10.1007/978-0-387-30164-8 · Cited by 3,465
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119
Proceedings for the 5th International Conference on Learning Representations
Srivastava, A., & Sutton, C. A. (2017) · Cited by 1,829
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815
Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents
Tan, M. (1993) · Machine Learning Proceedings 1993 · doi:10.1016/b978-1-55860-307-3.50049-6 · Cited by 1,810
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
Dietterich, T. G. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.639 · Cited by 1,435
Proceedings of the 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems
(2001) · Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems Expanding the Societal Role of Robotics in the the Next Millennium (Cat No 01CH37180) IROS-01 · doi:10.1109/iros.2001.977143 · Cited by 1,197
Dynamic Programming Treatment of the Travelling Salesman Problem
Bellman, R. (1962) · Journal of the ACM · doi:10.1145/321105.321111 · Cited by 808
Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI'12)
Hoffmann, J., & Selman, B. (2012) · HAL · Cited by 754
Chen, Z., & Liu, B. (2018) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.1007/978-3-031-01581-6 · Cited by 466
Learning to Walk Via Deep Reinforcement Learning
Haarnoja, T., Ha, S., Zhou, A., Tan, J., Tucker, G., & Levine, S. (2019) · Robotics: Science and Systems XV · doi:10.15607/rss.2019.xv.011 · Cited by 430
Recent Advances in Hierarchical Reinforcement Learning
Barto, A. G., & Mahadevan, S. (2003) · Discrete Event Dynamic Systems · doi:10.1023/a:1025696116075 · Cited by 317
Learning Options in Reinforcement Learning
Stolle, M., & Precup, D. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-45622-8_16 · Cited by 261
Transfer in Reinforcement Learning: A Framework and a Survey
Lazaric, A. (2012) · Adaptation, Learning, and Optimization · doi:10.1007/978-3-642-27645-3_5 · Cited by 256
Identifying useful subgoals in reinforcement learning by local graph partitioning
Şimşek, Ö., Wolfe, A. P., & Barto, A. G. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102454 · Cited by 238
Q-Cut—Dynamic Discovery of Sub-goals in Reinforcement Learning
Menache, I., Mannor, S., & Shimkin, N. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-36755-1_25 · Cited by 218
Hierarchical Deep Reinforcement Learning for Continuous Action Control
Yang, Z., Merrick, K., Jin, L., & Abbass, H. A. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2805379 · Cited by 197
Hierarchical multi-agent reinforcement learning
Makar, R., Mahadevan, S., & Ghavamzadeh, M. (2001) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/375735.376302 · Cited by 155
Chen, Z., & Liu, B. (2018) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.2200/s00737ed1v01y201610aim033 · Cited by 154
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
Gupta, A., Kumar, V., Lynch, C., Levine, S., & Hausman, K. (2019) · arXiv · doi:10.48550/arxiv.1910.11956 · Cited by 153 · cited 1×
“HRL algorithms have been shown to outperform standard RL in several long-horizon problems such as continuous control(2) [25, 54, 69], long-horizon games [51, 99], robot manipulation, [28, 33] and so on.”
MethodHierarchical multi-agent reinforcement learning
Ghavamzadeh, M., Mahadevan, S., & Makar, R. (2006) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-006-7035-4 · Cited by 141
Probabilistic inference for determining options in reinforcement learning
Daniel, C., van Hoof, H., Peters, J., & Neumann, G. M. (2016) · Machine Learning · doi:10.1007/s10994-016-5580-x · Cited by 95
Hierarchical Reinforcement Learning - A Survey
Al‐Emran, M. (2015) · International Journal of Computing and Digital Systems · doi:10.12785/ijcds/040207 · Cited by 50
The utility of temporal abstraction in reinforcement learning
Jong, N. K., Hester, T., & Stone, P. (2008) · doi:10.5555/1402383.1402429 · Cited by 45
Unified Inter and Intra Options Learning Using Policy Gradient Methods
Levy, K. Y., & Shimkin, N. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-29946-9_17 · Cited by 26
Options of Interest: Temporal Abstraction with Interest Functions
Khetarpal, K., Klissarov, M., Chevalier-Boisvert, M., Bacon, P., & Precup, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5871 · Cited by 23
Graph-Based Skill Acquisition For Reinforcement Learning
Mendonça, M. R. F., Ziviani, A., & Barreto, A. M. S. (2019) · ACM Computing Surveys · doi:10.1145/3291045 · Cited by 19
Hierarchical Reinforcement Learning
Hengst, B. (2010) · Encyclopedia of Machine Learning · doi:10.1007/978-0-387-30164-8_363 · Cited by 19
Cited by
Reinforcement learning algorithms: A brief survey
Shakya, A. K., Pillai, G., & Chakrabarty, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120495 · Cited by 656
Deep learning: systematic review, models, challenges, and research directions
Talaei Khoei, T., Ould Slimane, H., & Kaabouch, N. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08957-4 · Cited by 411
Deep reinforcement learning in smart manufacturing: A review and prospects
Li, C., Zheng, P., Yin, Y., Wang, B., & Wang, L. (2023) · CIRP Journal of Manufacturing Science and Technology · doi:10.1016/j.cirpj.2022.11.003 · Cited by 327
Agentic AI: A Comprehensive Survey of Technologies, Applications, and Societal Implications
Pati, A. K. (2025) · IEEE Access · doi:10.1109/access.2025.3585609 · Cited by 100
Hua, M., Qi, X., Chen, D., Jiang, K., Liu, Z. E., Sun, H., Zhou, Q., & Xu, H. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3574280 · Cited by 88
Metadata
- Authors
- Pateria, S., Subagdja, B., Tan, A.-H., & Quek, C.
- Venue
- ACM Computing Surveys
- Published
- 2021-06-05
- Volume
- 54
- Issue
- 5
- Pages
- 1-35
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3168892396
- Cited by
- 461
- References
- 36
- FWCI
- 18.08
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.