CoChat

Hierarchical Reinforcement Learning: A Survey and Open Research Challenges

Open access
Hutsebaut-Buysse, Matthias, Mets, Kevin, Latré, Steven · 2022 · Machine Learning and Knowledge Extraction

The DOI matches its Crossref record; 210 of the 210 references it cites were found in a registry or an index.

Abstract

Reinforcement learning (RL) allows an agent to solve sequential decision-making problems by interacting with an environment in a trial-and-error fashion. When these environments are very complex, pure random exploration of possible solutions often fails, or is very sample inefficient, requiring an unreasonable amount of interaction with the environment. Hierarchical reinforcement learning (HRL) utilizes forms of temporal- and state-abstractions in order to tackle these challenges, while simultaneously paving the road for behavior reuse and increased interpretability of RL systems. In this survey paper we first introduce a selection of problem-specific approaches, which provided insight in how to utilize often handcrafted abstractions in specific task settings. We then introduce the Options framework, which provides a more generic approach, allowing abstractions to be discovered and learned semi-automatically. Afterwards we introduce the goal-conditional approach, which allows sub-behaviors to be embedded in a continuous space. In order to further advance the development of HRL agents, capable of simultaneously learning abstractions and how to use them, solely from interaction with complex high dimensional environments, we also identify a set of promising research directions.

Overview

Integrity signals

Registry

Crossref · 10.3390/make4010009

Cites

210 of 210 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 210 references

Research works printed but not linked: 53

  1. Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press.
  2. Li, Y. (2017). Deep Reinforcement Learning: An Overview. arXiv.
  3. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv.
  4. Badia, A.P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, D., and Blundell, C. (2020). Agent57: Outperforming the Atari Human Benchmark. arXiv.
  5. OpenAI (2019). Dota 2 with Large Scale Deep Reinforcement Learning. arXiv.
  6. OpenAI (2019). Learning Dexterous In-Hand Manipulation. arXiv.
  7. Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., and Vanhoucke, V. (2018). QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. arXiv.
  8. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete problems in AI safety. arXiv.
  9. Burda, Y., Edwards, H., Storkey, A., and Klimov, O. (2018). Exploration by Random Network Distillation. arXiv.
  10. Bellemare, M.G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. (2016). Unifying Count-Based Exploration and Intrinsic Motivation. arXiv.
  11. Nachum, O., Tang, H., Lu, X., Gu, S., Lee, H., and Levine, S. (2019). Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?. arXiv.
  12. Mott, A., Zoran, D., Chrzanowski, M., Wierstra, D., and Rezende, D.J. (2019). Towards Interpretable Reinforcement Learning Using Attention Augmented Agents. arXiv.
  13. Garnelo, M., Arulkumaran, K., and Shanahan, M. (2016). Towards Deep Symbolic Reinforcement Learning. arXiv.
  14. Houthooft, R., Chen, X., Chen, X., Duan, Y., Schulman, J., Turck, F.D., and Abbeel, P. (2016). VIME: Variational Information Maximizing Exploration. arXiv.
  15. Harutyunyan, A., Dabney, W., Mesnard, T., Heess, N., Azar, M.G., Piot, B., van Hasselt, H., Singh, S., Wayne, G., and Precup, D. (2019). Hindsight Credit Assignment. arXiv.
  16. Rummery, G.A., and Niranjan, M. (1994). On-Line Q-Learning Using Connectionist Systems, University of Cambridge, Department of Engineering.
  17. Donahue, J., and Simonyan, K. (2019). Large Scale Adversarial Representation Learning. arXiv.
  18. Radford, A., Metz, L., and Chintala, S. (2016). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv.
  19. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016). WaveNet: A Generative Model for Raw Audio. arXiv.
  20. Sutskever, I., Vinyals, O., and Le, Q.V. (2014). Sequence to Sequence Learning with Neural Networks. arXiv.
  21. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv.
  22. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention Is All You Need. arXiv.
  23. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative Adversarial Nets. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
  24. Van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. (2018). Deep Reinforcement Learning and the Deadly Triad. arXiv.
  25. Zhang, S., and Sutton, R.S. (2017). A Deeper Look at Experience Replay. arXiv.
  26. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016). Prioritized Experience Replay. arXiv.
  27. Dabney, W., Rowland, M., Bellemare, M., and Munos, R. (2018, January 2–7). Distributional Reinforcement Learning with Quantile Regression. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA. · DOI · Paper page
  28. Ferns, N., Panangaden, P., and Precup, D. (2004, January 9–11). Metrics for Finite Markov Decision Processes. Proceedings of the Twentieth Conference on Uncertainty in Artificial Intelligence, Banff, AB, Canada.
  29. McGovern, A., Sutton, R.S., and Fagg, A.H. (1997, January 19–21). Roles of macro-actions in accelerating reinforcement learning. Proceedings of the Grace Hopper Celebration of Women in Computing, San Jose, CA, USA.
  30. Heess, N., Wayne, G., Tassa, Y., Lillicrap, T., Riedmiller, M., and Silver, D. (2016). Learning and transfer of modulated locomotor controllers. arXiv.
  31. Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. (2019). Diversity Is All You Need: Learning Skills without a Reward Function. arXiv.
  32. Abel, D., Umbanhowar, N., Khetarpal, K., Arumugam, D., Precup, D., and Littman, M.L. (2020, January 26–28). Value Preserving State-Action Abstractions. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, Palermo, Italy.
  33. Fox, R., Krishnan, S., Stoica, I., and Goldberg, K. (2017). Multi-level discovery of deep options. arXiv.
  34. Ng, A.Y., and Russell, S.J. (July, January 29). Algorithms for Inverse Reinforcement Learning. Proceedings of the International Conference on Machine Learning, Stanford, CA, USA.
  35. Pan, X., Ohn-Bar, E., Rhinehart, N., Xu, Y., Shen, Y., and Kitani, K.M. (2018). Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning. arXiv.
  36. Krishnan, S., Garg, A., Liaw, R., Miller, L., Pokorny, F.T., and Goldberg, K. (2016). Hirl: Hierarchical inverse reinforcement learning for long-horizon tasks with delayed rewards. arXiv.
  37. Kumar, A., Swersky, K., and Hinton, G. (2017, January 9). Feudal Learning for Large Discrete Action Spaces with Recursive Substructure. Proceedings of the NIPS Workshop Hierarchical Reinforcement Learning, Long Beach, CA, USA.
  38. Kulkarni, T.D., Saeedi, A., Gautam, S., and Gershman, S.J. (2016). Deep successor reinforcement learning. arXiv.
  39. Lakshminarayanan, A.S., Krishnamurthy, R., Kumar, P., and Ravindran, B. (2016). Option discovery in hierarchical reinforcement learning using spatio-temporal clustering. arXiv.
  40. Da Silva, B.C., Konidaris, G., and Barto, A.G. (July, January 26). Learning Parameterized Skills. Proceedings of the International Conference on Machine Learning, Edinburgh, Scotland.
  41. Wang, J.X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J.Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. (2017). Learning to Reinforcement Learn. arXiv.
  42. Duan, Y., Schulman, J., Chen, X., Bartlett, P.L., Sutskever, I., and Abbeel, P. (2016). RL$2^$: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv.
  43. Li, A.C., Florensa, C., Clavera, I., and Abbeel, P. (2020). Sub-Policy Adaptation for Hierarchical Reinforcement Learning. arXiv.
  44. Sohn, S., Woo, H., Choi, J., and Lee, H. (2020). Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies. arXiv.
  45. Bengio, E., Thomas, V., Pineau, J., Precup, D., and Bengio, Y. (2017). Independently Controllable Features. arXiv.
  46. Van Seijen, H., Fatemi, M., Romoff, J., Laroche, R., Barnes, T., and Tsang, J. (2017). Hybrid reward architecture for reinforcement learning. arXiv.
  47. Sukhbaatar, S., Denton, E., Szlam, A., and Fergus, R. (2018). Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning. arXiv.
  48. Gregor, K., Rezende, D.J., and Wierstra, D. (2016). Variational Intrinsic Control. arXiv.
  49. Achiam, J., Edwards, H., Amodei, D., and Abbeel, P. (2018). Variational option discovery algorithms. arXiv.
  50. Beattie, C., Leibo, J.Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., and Sadik, A. (2016). Deepmind lab. arXiv.
  51. Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets, A.S., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., and Schrittwieser, J. (2017). Starcraft ii: A new challenge for reinforcement learning. arXiv.
  52. Osband, I., Doron, Y., Hessel, M., Aslanides, J., Sezener, E., Saraiva, A., McKinney, K., Lattimore, T., Szepezvari, C., and Singh, S. (2019). Behaviour Suite for Reinforcement Learning. arXiv.
  53. Shang, W., Trott, A., Zheng, S., Xiong, C., and Socher, R. (2019). Learning World Graphs to Accelerate Hierarchical Reinforcement Learning. arXiv.

Cited by

See all 151

Metadata

Authors
Hutsebaut-Buysse, M., Mets, K., & Latré, S.
Affiliations
University of Antwerp (BE)
Corresponding author
Hutsebaut-Buysse, Matthias
Venue
Machine Learning and Knowledge Extraction
Publisher
MDPI AG
Published
2022-02-17
Volume
4
Issue
1
Pages
172-221
ISSN
2504-4990 (ISSN-L)
OpenAlex
W4212996499
Cited by
129
References
208
Citations by year
2026: 30 · 2025: 45 · 2024: 29 · 2023: 19 · 2022: 6
FWCI
8.15
Type
journal article
Language
en
Open access
Locations
  • Machine Learning and Knowledge Extraction (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
University of Antwerp (x)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
Keywords
hierarchical reinforcement learning, abstraction, temporal abstraction, state abstraction, options framework, goal-conditioned policies, sample efficiency, exploration, interpretability, high-dimensional environments, sequential decision making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Hierarchical Reinforcement Learning: A Survey and Open Research Challenges. https://app.cochat.ai/papers/10.3390/make4010009
The paper
Hutsebaut-Buysse, M., Mets, K., & Latré, S. (2022). Hierarchical Reinforcement Learning: A Survey and Open Research Challenges. Machine Learning and Knowledge Extraction. https://doi.org/10.3390/make4010009

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.