Hierarchical Reinforcement Learning: A Survey and Open Research Challenges
Open accessThe DOI matches its Crossref record; 210 of the 210 references it cites were found in a registry or an index.
Abstract
Reinforcement learning (RL) allows an agent to solve sequential decision-making problems by interacting with an environment in a trial-and-error fashion. When these environments are very complex, pure random exploration of possible solutions often fails, or is very sample inefficient, requiring an unreasonable amount of interaction with the environment. Hierarchical reinforcement learning (HRL) utilizes forms of temporal- and state-abstractions in order to tackle these challenges, while simultaneously paving the road for behavior reuse and increased interpretability of RL systems. In this survey paper we first introduce a selection of problem-specific approaches, which provided insight in how to utilize often handcrafted abstractions in specific task settings. We then introduce the Options framework, which provides a more generic approach, allowing abstractions to be discovered and learned semi-automatically. Afterwards we introduce the goal-conditional approach, which allows sub-behaviors to be embedded in a continuous space. In order to further advance the development of HRL agents, capable of simultaneously learning abstractions and how to use them, solely from interaction with complex high dimensional environments, we also identify a set of promising research directions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Pretraining is a common approach to increase the stability of the policy in hierarchical reinforcement learning [24].”
From Learning Hierarchical Interactive Multi-Object Search for Mobile Manipulation (2023) · cites this paper as [24]
Citations
Cites
Items 201–210 of 210
The Option Keyboard: Combining Skills in Reinforcement Learning
Barreto, A. D. M. S., Borsa, D., Hou, S., Comanici, G., Aygün, E., Hamel, P., Toyama, D., Hunt, J. J., Mourad, S., Silver, D., et al. (2019) · arXiv (Cornell University) · doi:10.13140/rg.2.2.12370.50882 · Cited by 35
Learnings Options End-to-End for Continuous Action TasksPreprint
Klissarov, M., Bacon, P., Harb, J., & Precup, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00004 · Cited by 32
Optimal policy switching algorithms for reinforcement learning
Comanici, G., & Precup, D. (2010) · Adaptive Agents and Multi-Agents Systems · doi:10.5555/1838206.1838300 · Cited by 30
Using Options and Covariance Testing for Long Horizon Off-Policy Policy EvaluationPreprint
Guo, Z. D., Thomas, P. S., & Brunskill, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.03453 · Cited by 29
Using Bisimulation for Policy Transfer in MDPs
Castro, P. F., & Precup, D. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7751 · Cited by 28
Continuous-Time Hierarchical Reinforcement Learning
Ghavamzadeh, M., & Mahadevan, S. (2001) · Cited by 27
Time-regularized interrupting options
Mankowitz, D. J., Mann, T., & Mannor, S. (2014) · Cited by 17
Finding Options that Minimize Planning TimePreprint
Jinnai, Y., Abel, D. L., Hershkowitz, D. E., Littman, M. L., & Konidaris, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.07311 · Cited by 14
Automatic Construction of Temporally Extended Actions for MDPs Using Bisimulation Metrics
Castro, P. S., & Precup, D. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-29946-9_16 · Cited by 10
Escape Room: A Configurable Testbed for Hierarchical Reinforcement Learning
Menashe, J., & Stone, P. (2019) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/juvy9794 · Cited by 5
Research works printed but not linked: 53
- Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press.
- Li, Y. (2017). Deep Reinforcement Learning: An Overview. arXiv.
- Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv.
- Badia, A.P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, D., and Blundell, C. (2020). Agent57: Outperforming the Atari Human Benchmark. arXiv.
- OpenAI (2019). Dota 2 with Large Scale Deep Reinforcement Learning. arXiv.
- OpenAI (2019). Learning Dexterous In-Hand Manipulation. arXiv.
- Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., and Vanhoucke, V. (2018). QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. arXiv.
- Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete problems in AI safety. arXiv.
- Burda, Y., Edwards, H., Storkey, A., and Klimov, O. (2018). Exploration by Random Network Distillation. arXiv.
- Bellemare, M.G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. (2016). Unifying Count-Based Exploration and Intrinsic Motivation. arXiv.
- Nachum, O., Tang, H., Lu, X., Gu, S., Lee, H., and Levine, S. (2019). Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?. arXiv.
- Mott, A., Zoran, D., Chrzanowski, M., Wierstra, D., and Rezende, D.J. (2019). Towards Interpretable Reinforcement Learning Using Attention Augmented Agents. arXiv.
- Garnelo, M., Arulkumaran, K., and Shanahan, M. (2016). Towards Deep Symbolic Reinforcement Learning. arXiv.
- Houthooft, R., Chen, X., Chen, X., Duan, Y., Schulman, J., Turck, F.D., and Abbeel, P. (2016). VIME: Variational Information Maximizing Exploration. arXiv.
- Harutyunyan, A., Dabney, W., Mesnard, T., Heess, N., Azar, M.G., Piot, B., van Hasselt, H., Singh, S., Wayne, G., and Precup, D. (2019). Hindsight Credit Assignment. arXiv.
- Rummery, G.A., and Niranjan, M. (1994). On-Line Q-Learning Using Connectionist Systems, University of Cambridge, Department of Engineering.
- Donahue, J., and Simonyan, K. (2019). Large Scale Adversarial Representation Learning. arXiv.
- Radford, A., Metz, L., and Chintala, S. (2016). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv.
- van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016). WaveNet: A Generative Model for Raw Audio. arXiv.
- Sutskever, I., Vinyals, O., and Le, Q.V. (2014). Sequence to Sequence Learning with Neural Networks. arXiv.
- Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention Is All You Need. arXiv.
- Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative Adversarial Nets. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
- Van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. (2018). Deep Reinforcement Learning and the Deadly Triad. arXiv.
- Zhang, S., and Sutton, R.S. (2017). A Deeper Look at Experience Replay. arXiv.
- Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016). Prioritized Experience Replay. arXiv.
- Dabney, W., Rowland, M., Bellemare, M., and Munos, R. (2018, January 2–7). Distributional Reinforcement Learning with Quantile Regression. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA. · DOI · Paper page
- Ferns, N., Panangaden, P., and Precup, D. (2004, January 9–11). Metrics for Finite Markov Decision Processes. Proceedings of the Twentieth Conference on Uncertainty in Artificial Intelligence, Banff, AB, Canada.
- McGovern, A., Sutton, R.S., and Fagg, A.H. (1997, January 19–21). Roles of macro-actions in accelerating reinforcement learning. Proceedings of the Grace Hopper Celebration of Women in Computing, San Jose, CA, USA.
- Heess, N., Wayne, G., Tassa, Y., Lillicrap, T., Riedmiller, M., and Silver, D. (2016). Learning and transfer of modulated locomotor controllers. arXiv.
- Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. (2019). Diversity Is All You Need: Learning Skills without a Reward Function. arXiv.
- Abel, D., Umbanhowar, N., Khetarpal, K., Arumugam, D., Precup, D., and Littman, M.L. (2020, January 26–28). Value Preserving State-Action Abstractions. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, Palermo, Italy.
- Fox, R., Krishnan, S., Stoica, I., and Goldberg, K. (2017). Multi-level discovery of deep options. arXiv.
- Ng, A.Y., and Russell, S.J. (July, January 29). Algorithms for Inverse Reinforcement Learning. Proceedings of the International Conference on Machine Learning, Stanford, CA, USA.
- Pan, X., Ohn-Bar, E., Rhinehart, N., Xu, Y., Shen, Y., and Kitani, K.M. (2018). Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning. arXiv.
- Krishnan, S., Garg, A., Liaw, R., Miller, L., Pokorny, F.T., and Goldberg, K. (2016). Hirl: Hierarchical inverse reinforcement learning for long-horizon tasks with delayed rewards. arXiv.
- Kumar, A., Swersky, K., and Hinton, G. (2017, January 9). Feudal Learning for Large Discrete Action Spaces with Recursive Substructure. Proceedings of the NIPS Workshop Hierarchical Reinforcement Learning, Long Beach, CA, USA.
- Kulkarni, T.D., Saeedi, A., Gautam, S., and Gershman, S.J. (2016). Deep successor reinforcement learning. arXiv.
- Lakshminarayanan, A.S., Krishnamurthy, R., Kumar, P., and Ravindran, B. (2016). Option discovery in hierarchical reinforcement learning using spatio-temporal clustering. arXiv.
- Da Silva, B.C., Konidaris, G., and Barto, A.G. (July, January 26). Learning Parameterized Skills. Proceedings of the International Conference on Machine Learning, Edinburgh, Scotland.
- Wang, J.X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J.Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. (2017). Learning to Reinforcement Learn. arXiv.
- Duan, Y., Schulman, J., Chen, X., Bartlett, P.L., Sutskever, I., and Abbeel, P. (2016). RL$2^$: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv.
- Li, A.C., Florensa, C., Clavera, I., and Abbeel, P. (2020). Sub-Policy Adaptation for Hierarchical Reinforcement Learning. arXiv.
- Sohn, S., Woo, H., Choi, J., and Lee, H. (2020). Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies. arXiv.
- Bengio, E., Thomas, V., Pineau, J., Precup, D., and Bengio, Y. (2017). Independently Controllable Features. arXiv.
- Van Seijen, H., Fatemi, M., Romoff, J., Laroche, R., Barnes, T., and Tsang, J. (2017). Hybrid reward architecture for reinforcement learning. arXiv.
- Sukhbaatar, S., Denton, E., Szlam, A., and Fergus, R. (2018). Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning. arXiv.
- Gregor, K., Rezende, D.J., and Wierstra, D. (2016). Variational Intrinsic Control. arXiv.
- Achiam, J., Edwards, H., Amodei, D., and Abbeel, P. (2018). Variational option discovery algorithms. arXiv.
- Beattie, C., Leibo, J.Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., and Sadik, A. (2016). Deepmind lab. arXiv.
- Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets, A.S., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., and Schrittwieser, J. (2017). Starcraft ii: A new challenge for reinforcement learning. arXiv.
- Osband, I., Doron, Y., Hessel, M., Aslanides, J., Sezener, E., Saraiva, A., McKinney, K., Lattimore, T., Szepezvari, C., and Singh, S. (2019). Behaviour Suite for Reinforcement Learning. arXiv.
- Shang, W., Trott, A., Zheng, S., Xiong, C., and Socher, R. (2019). Learning World Graphs to Accelerate Hierarchical Reinforcement Learning. arXiv.
Cited by
Al-Hamadani, M. N. A., Fadhel, M. A., Alzubaidi, L., & Harangi, B. (2024) · Sensors · doi:10.3390/s24082461 · Cited by 123
A sensory–motor theory of the neocortex
Rao, R. P. N. (2024) · Nature Neuroscience · doi:10.1038/s41593-024-01673-9 · Cited by 103
Reinforcement learning for predictive maintenance: a systematic technical review
Siraskar, R., Kumar, S., Patil, S., Bongale, A. M., & Kotecha, K. V. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10468-6 · Cited by 92
Deep reinforcement learning challenges and opportunities for urban water systems
Negm, A. S., Ma, X., & Aggidis, G. (2024) · Water Research · doi:10.1016/j.watres.2024.121145 · Cited by 70
Reinforcement learning for vehicle-to-grid: A review
Xie, H., Song, G., Shi, Z., Zhang, J., Lin, Z., Yu, Q., Fu, H., Song, X., & Zhang, H. (2025) · Advances in Applied Energy · doi:10.1016/j.adapen.2025.100214 · Cited by 81
Metadata
- Authors
- Hutsebaut-Buysse, M., Mets, K., & Latré, S.
- Affiliations
- University of Antwerp (BE)
- Corresponding author
- Hutsebaut-Buysse, Matthias
- Venue
- Machine Learning and Knowledge Extraction
- Publisher
- MDPI AG
- Published
- 2022-02-17
- Volume
- 4
- Issue
- 1
- Pages
- 172-221
- ISSN
- 2504-4990 (ISSN-L)
- OpenAlex
- W4212996499
- Cited by
- 129
- References
- 208
- Citations by year
- 2026: 30 · 2025: 45 · 2024: 29 · 2023: 19 · 2022: 6
- FWCI
- 8.15
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Machine Learning and Knowledge Extraction (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- University of Antwerp (x)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Keywords
- hierarchical reinforcement learning, abstraction, temporal abstraction, state abstraction, options framework, goal-conditioned policies, sample efficiency, exploration, interpretability, high-dimensional environments, sequential decision making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.