Hierarchical Reinforcement Learning: A Survey and Open Research Challenges
Open accessThe DOI matches its Crossref record; 210 of the 210 references it cites were found in a registry or an index.
Abstract
Reinforcement learning (RL) allows an agent to solve sequential decision-making problems by interacting with an environment in a trial-and-error fashion. When these environments are very complex, pure random exploration of possible solutions often fails, or is very sample inefficient, requiring an unreasonable amount of interaction with the environment. Hierarchical reinforcement learning (HRL) utilizes forms of temporal- and state-abstractions in order to tackle these challenges, while simultaneously paving the road for behavior reuse and increased interpretability of RL systems. In this survey paper we first introduce a selection of problem-specific approaches, which provided insight in how to utilize often handcrafted abstractions in specific task settings. We then introduce the Options framework, which provides a more generic approach, allowing abstractions to be discovered and learned semi-automatically. Afterwards we introduce the goal-conditional approach, which allows sub-behaviors to be embedded in a continuous space. In order to further advance the development of HRL agents, capable of simultaneously learning abstractions and how to use them, solely from interaction with complex high dimensional environments, we also identify a set of promising research directions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Pretraining is a common approach to increase the stability of the policy in hierarchical reinforcement learning [24].”
From Learning Hierarchical Interactive Multi-Object Search for Mobile Manipulation (2023) · cites this paper as [24]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 1×
“Building blocks such as Long Short-Term Memory (LSTM) [107] recurrent networks and Gated Recurrent Units (GRU ([108] allow deep learning to work with data which is sequential in nature.”
MethodReducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“In order to do this, inspiration might be found in how convolutional neural networks are often visualized [254].”
BackgroundMastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
The graph neural network model
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2005605 · Cited by 9,582 · cited 1×
“For example, Graph Neural Networks (GNN) [114,115] are capable of directly working with data structured as a graph.”
BackgroundWatkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · cited 2×
“The UNsupervised REinforcement and Auxiliary Learning (UNREAL) architecture [225] uses pixel-control and feature-control as auxiliary rewards which can be optimized together with an A3C agent [226].”
MethodHow transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · cited 1×
“Drawing inspiration from research done in deep learning on how to transfer visual representations from one task to another [58,59], HRL methods have shown to be capable of learning transferable abstractions within the same problem setting [60].”
MethodMarkov Decision Processes: Discrete Stochastic Dynamic Programming
Puterman, M. L. (1994) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316887 · Cited by 6,024
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · cited 3×
“For example, the Scheduled Auxiliary Control (SAC-X) architecture [227] uses auxiliary tasks such as minimizing/maximizing distance between objects, maximizing velocity, or activating a touch-sensor in order to actively drive scheduled exploration.”
BackgroundMethodBengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 1×
“The DQN algorithm has been heavily studied, and various improvements have been proposed such as Double Q-Learning (DDQN) [122], prioritized replay [123], dueling networks [124], multi-step learning [125] and distributional RL [75,76,126].”
Method(1999) · Cited by 5,147
Deep learning in agriculture: A survey
Kamilaris, A., & Prenafeta-Boldú, F. X. (2018) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2018.02.016 · Cited by 5,133
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733
A Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 1×
“Other recent successes of deep RL include beating top professional human players in the complex board game of Go [14], and achieving human level performance in cooperative 3D multiplayer video games such as DOTA 2 [15,16], StarCraft II [17] and a modified version of Quake III Arena [18].”
BackgroundSimple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318
Dueling Network Architectures for Deep Reinforcement Learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06581 · Cited by 3,300
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,233
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119
Solving the multiple instance problem with axis-parallel rectangles
Dietterich, T. G., Lathrop, R. H., & Lozano-Pérez, T. (1997) · Artificial Intelligence · doi:10.1016/s0004-3702(96)00034-3 · Cited by 2,866 · cited 1×
“McGovern and Barto [154] described the problem of discovering bottleneck states, as an application of multiple-instance learning [191].”
BackgroundA Neural Algorithm of Artistic Style
Gatys, L., Ecker, A., & Bethge, M. (2016) · Journal of Vision · doi:10.1167/16.12.326 · Cited by 2,612
An Introduction to Variational Autoencoders
Kingma, D. P., & Welling, M. (2019) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000056 · Cited by 2,609 · cited 2×
“These techniques include Variational Autoencoders (VAE) [111], which utilize a regularized bottleneck autoencoder approach.”
MethodBuilding machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550
Building Machines That Learn and Think Like People
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2016) · arXiv · doi:10.48550/arxiv.1604.00289 · Cited by 2,550
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
Konda, V. R., & Tsitsiklis, J. N. (2002) · Cited by 2,202
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161 · cited 6×
“, Asteroids, Breakout, Pong) [118] has been the most widely used benchmark to demonstrate capabilities of hierarchical algorithms.”
BackgroundResultInfluentialA new model for learning in graph domains
Gori, M., Monfardini, G., & Scarselli, F. (2006) · Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005. · doi:10.1109/ijcnn.2005.1555942 · Cited by 2,075 · cited 1×
“For example, Graph Neural Networks (GNN) [114,115] are capable of directly working with data structured as a graph.”
BackgroundLevine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971
Generative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941
Maximum Entropy Inverse Reinforcement Learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2018) · doi:10.1184/r1/6555512 · Cited by 1,869
Rainbow: Combining Improvements in Deep Reinforcement Learning
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., & Silver, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11796 · Cited by 1,846
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815
DeepMimic: example-guided deep reinforcement learning of physics-based character skills
Peng, X. B., Abbeel, P., Levine, S., & van de Panne, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201311 · Cited by 1,784
Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping
Ng, A. Y., Harada, D., & Russell, S. (1999) · Cited by 1,744
Deep Learning of Representations for Unsupervised and Transfer Learning.
Bengio, Y. (2011) · Cited by 1,702
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685
Self-improving reactive agents based on reinforcement learning, planning and teaching
Lin, L.-J. (1992) · Machine Learning · doi:10.1007/bf00992699 · Cited by 1,564 · cited 1×
“Additionally, an experience replay buffer [120,121] was used, in order to reduce temporal correlation in the observed state-reward sequences.”
MethodHierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
Dietterich, T. G. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.639 · Cited by 1,435
Resource Management with Deep Reinforcement Learning
Mao, H., Alizadeh, M., Menache, I., & Kandula, S. (2016) · Proceedings of the 15th ACM Workshop on Hot Topics in Networks · doi:10.1145/3005745.3005750 · Cited by 1,353 · cited 1×
“autonomous vehicles [27,28] smart grids [29], fleet management [30], resource management [31,32] and recommender systems [33].”
BackgroundTree-Based Batch Mode Reinforcement Learning
Ernst, D., Geurts, P., & Wehenkel, L. A. (2005) · Journal of Machine Learning Research · doi:10.5555/1046920.1088690 · Cited by 1,351
End-to-End Training of Deep Visuomotor Policies
Levine, S., Finn, C., Darrell, T. J., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 1,344
wav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
An Introduction to Deep Reinforcement Learning
Vincent, F.-L., Peter, H., Riashat, I., Marc, G. B., & Joelle, P. (2018) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000071 · Cited by 1,286
A simple neural network module for relational reasoning
Santoro, A., Raposo, D., Barrett, D. G. T., Malinowski, M., Pascanu, R., Battaglia, P., & Lillicrap, T. (2017) · arXiv · doi:10.48550/arxiv.1706.01427 · Cited by 1,281 · cited 1×
“In order to scale to problems, which require very long-horizon planning, without dense feedback, it might be required to look at radical alternative approaches, such as representing sub-behaviors in a graph-structure [219,265] or taking into account relations between objects [83,266,267].”
BackgroundA Framework for Multiple-Instance Learning
Maron, O., & Lozano‐Pérez, T. (1997) · neural information processing systems · Cited by 1,230
Bisimulation through probabilistic testing
Larsen, K. G., & Skou, A. (1991) · Information and Computation · doi:10.1016/0890-5401(91)90030-6 · Cited by 1,206
Planning in a hierarchy of abstraction spaces
Sacerdoti, E. D. (1974) · Artificial Intelligence · doi:10.1016/0004-3702(74)90026-5 · Cited by 1,155
(2015) · Journal of Machine Learning Research · Cited by 1,119
Benchmarking Deep Reinforcement Learning for Continuous Control
Duan, Y., Chen, X., Houthooft, R., Schulman, J., & Abbeel, P. (2016) · arXiv · doi:10.48550/arxiv.1604.06778 · Cited by 1,114
End-to-end attention-based large vocabulary speech recognition
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472618 · Cited by 1,096
Deep convolutional neural networks for LVCSR
Sainath, T. N., Mohamed, A.-R., Kingsbury, B., & Ramabhadran, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639347 · Cited by 1,053
Recent Advances in Hierarchical Reinforcement Learning
Barto, A. G., & Mahadevan, S. (2003) · Discrete Event Dynamic Systems · doi:10.1023/a:1022140919877 · Cited by 1,024
On the theory of dynamic programming
Bellman, R. (1952) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.38.8.716 · Cited by 998
Reinforcement Learning in Continuous Time and Space
Doya, K. (2000) · Neural Computation · doi:10.1162/089976600300015961 · Cited by 960
DRAW: A Recurrent Neural Network For Image Generation
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.04623 · Cited by 841
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 770
Quantifying Generalization in Reinforcement Learning
Cobbe, K., Klimov, O., Hesse, C., Kim, T., & Schulman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.02341 · Cited by 769
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 749
Formal theory of creativity, fun, and intrinsic motivation (1990–2010)
Schmidhuber, J. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2056368 · Cited by 747
Universal Value Function Approximators
Schaul, T., Horgan, D., Gregor, K., & Silver, D. (2015) · Cited by 701
Improving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693
The Option-Critic Architecture
Bacon, P.-L., Harb, J., & Precup, D. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10916 · Cited by 687
Human-level performance in 3D multiplayer games with population-based reinforcement learning
Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castañeda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., et al. (2019) · Science · doi:10.1126/science.aau6249 · Cited by 662
Reinforcement Learning with Hierarchies of Machines
Parr, R., & Russell, S. (1997) · Cited by 618
Distributional Reinforcement Learning With Quantile Regression
Dabney, W., Rowland, M., Bellemare, M., & Munos, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11791 · Cited by 612
Automatic programming of behavior-based robots using reinforcement learning
Mahadevan, S., & Connell, J. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90058-6 · Cited by 591
Noisy Networks for Exploration
Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.10295 · Cited by 543
ViZDoom: A Doom-based AI research platform for visual reinforcement learning
Kempka, M., Wydmuch, M., Runc, G., Toczek, J., & Jaskowski, W. (2016) · 2016 IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2016.7860433 · Cited by 541
Dayan, P., & Hinton, G. E. (1992) · Cited by 540
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497
Relative Entropy Policy Search
Peters, J., Mulling, K., & Altun, Y. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7727 · Cited by 455
Data-efficient hierarchical reinforcement learning
Nachum, O., Gu, S., Lee, H., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.08296 · Cited by 446 · cited 11×
“Algorithms such as FeUdal Networks (FuN) [150] and HIRO [149] are capable of learning diverse sets of sub-behaviors and their composition, end-to-end, in function of the extrinsic reward.”
BackgroundMethodInfluentialA survey of real-time strategy game AI research and competition in StarCraft
Ontanon, S., Synnaeve, G., Uriarte, A., Richoux, F., Churchill, D., & Preuss, M. (2013) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2013.2286295 · Cited by 423
Automatic Discovery of Subgoals in Reinforcement Learning using Diverse Density
McGovern, A., & Barto, A. G. (2001) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 418
Large-Scale Study of Curiosity-Driven Learning
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1808.04355 · Cited by 401 · cited 2×
“Additionally, alternative forms of intrinsic reward signals, which are not specific to certain environments, such as (episodic) exploration bonuses [51,256], have not yet been fully researched in the context of automatic option-discovery, but could lead to more sample-efficient option-discovery methods because of thei…”
BackgroundImplementation Matters in Deep RL: A Case Study on PPO and TRPO
Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., & Mądry, A. (2020) · International Conference on Learning Representations · Cited by 398
Reinforcement Learning with Deep Energy-Based Policies
Haarnoja, T., Tang, H., Abbeel, P., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08165 · Cited by 392
PAC model-free reinforcement learning
Strehl, A. L., Li, L., Wiewiora, E., Langford, J., & Littman, M. L. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143955 · Cited by 384
Efficient Large-Scale Fleet Management via Multi-Agent Deep Reinforcement Learning
Lin, K., Zhao, R., Xu, Z., & Zhou, J. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219993 · Cited by 379
Learning to coordinate behaviors
Maes, P., & Brooks, R. A. (1990) · Cited by 376
Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining
Konidaris, G., & Barreto, A. S. (2009) · neural information processing systems · Cited by 366
A framework for behavioural cloning
Bain, M., & Sammut, C. (2000) · Machine Intelligence 15 · doi:10.1093/oso/9780198538677.003.0006 · Cited by 360
Parameter Space Noise for Exploration
Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., & Andrychowicz, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.01905 · Cited by 346
Learning an Embedding Space for Transferable Robot Skills
Hausman, K., Springenberg, J. T., Wang, Z., Heess, N., & Riedmiller, M. (2018) · International Conference on Learning Representations · Cited by 341
Linearly-solvable Markov decision problems
Todorov, E. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0176 · Cited by 334
Implicit Quantile Networks for Distributional Reinforcement Learning
Dabney, W., Ostrovski, G., Silver, D., & Munos, R. (2018) · arXiv · doi:10.48550/arxiv.1806.06923 · Cited by 331
Virtual to Real Reinforcement Learning for Autonomous Driving
Pan, X., You, Y., Wang, Z., & Lu, C. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.11 · Cited by 326 · cited 1×
“autonomous vehicles [27,28] smart grids [29], fleet management [30], resource management [31,32] and recommender systems [33].”
BackgroundVirtual to Real Reinforcement Learning for Autonomous Driving
Pan, X., You, Y., Wang, Z., & Lu, C. (2017) · arXiv · doi:10.48550/arxiv.1704.03952 · Cited by 326
A Deep Hierarchical Approach to Lifelong Learning in Minecraft
Tessler, C., Givony, S., Zahavy, T., Mankowitz, D., & Mannor, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10744 · Cited by 299
Towards a Unified Theory of State Abstraction for MDPs.
Li, L. C., Walsh, T. J., & Littman, M. L. (2006) · Cited by 293
(2013) · Cited by 290
Transfer of learning by composing solutions of elemental sequential tasks
Singh, S. P. (1992) · Machine Learning · doi:10.1007/bf00992700 · Cited by 288
Autonomous Helicopter Flight via Reinforcement Learning
Kim, H. J., Jordan, M. I., Sastry, S., & Ng, A. Y. (2003) · neural information processing systems · Cited by 287
Sutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M., White, A., & Precup, D. (2011) · doi:10.5555/2031678.2031726 · Cited by 283
Rusu, A. A., Colmenarejo, S. G., Gulcehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukcuoglu, K., & Hadsell, R. (2015) · arXiv · doi:10.48550/arxiv.1511.06295 · Cited by 270
Reinforcement Learning Methods for Continuous-Time Markov Decision Problems
Bradtke, S. J., & Duff, M. O. (1994) · Cited by 263
Learning Options in Reinforcement Learning
Stolle, M., & Precup, D. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-45622-8_16 · Cited by 261
Deep Learning for Video Game Playing
Justesen, N., Bontrager, P., Togelius, J., & Risi, S. (2020) · IEEE Transactions on Games · doi:10.1109/tg.2019.2896986 · Cited by 257
Reinforcement Learning with Unsupervised Auxiliary Tasks
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05397 · Cited by 247
Learning by Playing - Solving Sparse Reward Tasks from Scratch
Riedmiller, M., Hafner, R., Lampe, T., Neunert, M., Degrave, J., Van de Wiele, T., Mnih, V., Heess, N., & Springenberg, J. T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10567 · Cited by 242
Identifying useful subgoals in reinforcement learning by local graph partitioning
Şimşek, Ö., Wolfe, A. P., & Barto, A. G. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102454 · Cited by 238
A Distributional Perspective on Reinforcement Learning
Bellemare, M. G., Dabney, W., & Munos, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06887 · Cited by 237
FeUdal Networks for Hierarchical Reinforcement Learning
Vezhnevets, A., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., & Kavukcuoglu, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01161 · Cited by 233
An object-oriented representation for efficient reinforcement learning
Diuk, C., Cohen, A., & Littman, M. L. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390187 · Cited by 226
The Ingredients of Real-World Robotic Reinforcement LearningPreprint
Zhu, H., Yu, J., Gupta, A., Shah, D., Hartikainen, K., Singh, A., Kumar, V., & Levine, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12570 · Cited by 222
Hierarchical Imitation and Reinforcement LearningPreprint
Le, H. M., Jiang, N., Agarwal, A., Dudík, M., Yue, Y., & Daumé, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.00590 · Cited by 221
Q-Cut—Dynamic Discovery of Sub-goals in Reinforcement Learning
Menache, I., Mannor, S., & Shimkin, N. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-36755-1_25 · Cited by 218
The Malmo platform for artificial intelligence experimentation
Johnson, M., Hofmann, K., Hutton, T. J., & Bignell, D. E. (2016) · International Joint Conference on Artificial Intelligence · Cited by 210
Hierarchical Reinforcement Learning for Course Recommendation in MOOCs
Zhang, J., Hao, B., Chen, B., Li, C., Chen, H., & Sun, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301435 · Cited by 196 · cited 1×
“autonomous vehicles [27,28] smart grids [29], fleet management [30], resource management [31,32] and recommender systems [33].”
BackgroundSample Efficient Actor-Critic with Experience Replay
Wang, Z., Bapst, V., Heess, N., Mnih, V., Munos, R., Kavukcuoglu, K., & de Freitas, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01224 · Cited by 192
A Survey of Reinforcement Learning Informed by Natural Language
Luketina, J., Nardelli, N., Farquhar, G., Foerster, J., Andreas, J., Grefenstette, E., Whiteson, S., & Rocktäschel, T. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/880 · Cited by 188
HTN planning: Overview, comparison, and beyond
Georgievski, I., & Aiello, M. (2015) · Artificial Intelligence · doi:10.1016/j.artint.2015.02.002 · Cited by 187
Discovering Hierarchy in Reinforcement Learning with HEXQ
Hengst, B. (2002) · Cited by 181
Stochastic Neural Networks for Hierarchical Reinforcement Learning
Florensa, C., Duan, Y., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03012 · Cited by 180
Salge, C., Glackin, C., & Polani, D. (2013) · Emergence, complexity and computation · doi:10.1007/978-3-642-53734-9_4 · Cited by 178
Using relative novelty to identify useful temporal abstractions in reinforcement learning
Şimşek, Ö., & Barto, A. G. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015353 · Cited by 176
Q-PrOP: Sample-efficient policy gradient with an off-policy critic
Gu, S., Lillicrap, T., Ghahramani, Z., Turner, R. E., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.17863/cam.21294 · Cited by 174
Hierarchical Learning in Stochastic Domains: Preliminary Results
Kaelbling, L. P. (1993) · Machine Learning Proceedings 1993 · doi:10.1016/b978-1-55860-307-3.50028-9 · Cited by 172
Episodic Curiosity through Reachability
Savinov, N., Raichuk, A., Marinier, R., Vincent, D., Pollefeys, M., Lillicrap, T., & Gelly, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02274 · Cited by 170
Hierarchical Relative Entropy Policy Search
Daniel, C., Neumann, G. M., Kroemer, O., & Peters, J. (2014) · Lincoln Repository (University of Lincoln) · Cited by 166
Visualizing and Understanding Atari Agents
Greydanus, S., Koul, A., Dodge, J. E., & Fern, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00138 · Cited by 155
Decomposition Techniques for Planning in Stochastic Domains
Dean, T., & Lin, S. (1995) · International Joint Conference on Artificial Intelligence · Cited by 154
Towards Sample Efficient Reinforcement Learning
Yu, Y. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/820 · Cited by 147
Analysis of Temporal-Diffference Learning with Function Approximation
Tsitsiklis, J. N., & Roy, B. V. (1996) · neural information processing systems · Cited by 145
Deep reinforcement learning with relational inductive biases
Zambaldi, V., Raposo, D. C., Santoro, A., Bapst, V., Li, Y., Babuschkin, I., Tuyls, K., Reichert, D., Lillicrap, T., Lockhart, E., et al. (2018) · International Conference on Learning Representations · Cited by 145
Diagnosing Bottlenecks in Deep Q-learning AlgorithmsPreprint
Fu, J., Kumar, A., Soh, M., & Levine, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10250 · Cited by 144
Intra-Option Learning about Temporally Abstract Actions
Sutton, R. S., Precup, D., & Singh, S. P. (1998) · Cited by 129
Deep Reinforcement Learning Solutions for Energy Microgrids Management
François-Lavet, V., Taralla, D., Ernst, D., & Raphael Fonteneau, D. (2016) · ORBi (University of Liège) · Cited by 128
Learning Stochastic Feedforward Neural Networks
Tang, Y., & Salakhutdinov, R. (2013) · neural information processing systems · Cited by 126
Learning Actionable Representations with Goal-Conditioned PoliciesPreprint
Ghosh, D., Gupta, A., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.07819 · Cited by 113
Language as an Abstraction for Hierarchical Deep Reinforcement Learning
Jiang, Y., Gu, S., Murphy, K., & Finn, C. (2019) · arXiv · doi:10.48550/arxiv.1906.07343 · Cited by 109
Offline policy evaluation across representations with applications to educational games
Mandel, T., Liu, Y.-E., Levine, S., Brunskill, E., & Popović, Z. (2014) · Adaptive Agents and Multi-Agents Systems · doi:10.5555/2615731.2617417 · Cited by 108
A Hierarchical Model for Device Placement
Mirhoseini, A., Goldie, A., Pham, H., Steiner, B., Le, Q. V., & Dean, J. (2018) · International Conference on Learning Representations · Cited by 108
Multi-time Models for Temporally Abstract Planning
Precup, D., & Sutton, R. S. (1997) · Cited by 108
Mahadevan, S. (2005) · International Conference on Machine Learning · doi:10.1145/1102351.1102421 · Cited by 104
Meta Learning Shared Hierarchies
Frans, K., Ho, J., Chen, X., Abbeel, P., & Schulman, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.09767 · Cited by 103
Learning Hierarchical Control Structures for Multiple Tasks and Changing Environments
Digney, B. L. (1998) · The MIT Press eBooks · doi:10.7551/mitpress/3119.003.0050 · Cited by 95
Probabilistic inference for determining options in reinforcement learning
Daniel, C., van Hoof, H., Peters, J., & Neumann, G. M. (2016) · Machine Learning · doi:10.1007/s10994-016-5580-x · Cited by 95
PAC-inspired Option Discovery in Lifelong Reinforcement Learning
Brunskill, E., & Li, L. (2014) · Cited by 79
Dynamic Action Repetition for Deep Reinforcement Learning
Lakshminarayanan, A. S., Sharma, S., & Ravindran, B. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10918 · Cited by 78
Strategic Attentive Writer for Learning Macro-Actions
-, A., Vezhnevets, Mnih, V., Agapiou, J., Osindero, S., Graves, A., Vinyals, O., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04695 · Cited by 78
Causal Graph Based Decomposition of Factored MDPs
Jönsson, A., & Barto, A. G. (2006) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.5555/1248547.1248628 · Cited by 77
Automatic discovery and transfer of MAXQ hierarchies
Mehta, N., Ray, S., Tadepalli, P. V., & Dietterich, T. G. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390238 · Cited by 76
Reinforcement Learning in Continuous Time and Space: A Stochastic Control Approach
Wang, H., Zariphopoulou, T., & Zhou, X. Y. (2020) · Journal of Machine Learning Research · Cited by 76
When Waiting Is Not an Option: Learning Options With a Deliberation Cost
Harb, J., Bacon, P.-L., Klissarov, M., & Precup, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11831 · Cited by 74
A Laplacian Framework for Option Discovery in Reinforcement Learning
Machado, M. C., Bellemare, M. G., & Bowling, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00956 · Cited by 72
Efficient skill learning using abstraction selection
Konidaris, G., & Barto, A. G. (2009) · Cited by 69
Evaluation of a hierarchical reinforcement learning spoken dialogue system
Cuayáhuitl, H., Renals, S. J., Lemon, O., & Shimodaira, H. (2009) · Computer Speech & Language · doi:10.1016/j.csl.2009.07.001 · Cited by 68
Self-Improving Factory Simulation using Continuous-time Average-Reward Reinforcement Learning
Mahadevan, S., Das, T. K., & Gosavi, A. (2007) · Cited by 67
Eigenoption Discovery through the Deep Successor Representation
Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G. J., & Campbell, M. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.11089 · Cited by 66
Hierarchical Reinforcement Learning with HindsightPreprint
Levy, A., Platt, R., & Saenko, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08180 · Cited by 66
Real-Time Reinforcement LearningPreprint
Ramstedt, S., & Pal, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04448 · Cited by 66
Latent Space Policies for Hierarchical Reinforcement Learning
Haarnoja, T., Hartikainen, K., Abbeel, P., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.02808 · Cited by 64
Exploration in Reinforcement Learning with Deep Covering Options
Jinnai, Y., Park, J. W., Machado, M. C., & Konidaris, G. (2020) · International Conference on Learning Representations · Cited by 60
Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
Nachum, O., Gu, S., Lee, H., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.01257 · Cited by 57
An Introduction to Intertask Transfer for Reinforcement Learning
Taylor, M. E., & Stone, P. (2011) · AI Magazine · doi:10.1609/aimag.v32i1.2329 · Cited by 55
Sample-Efficient Deep Reinforcement Learning via Episodic Backward UpdatePreprint
Lee, S. Y., Choi, S., & Chung, S.-Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.12375 · Cited by 55
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning
Shu, T., Xiong, C., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.07294 · Cited by 54
DDCO: Discovery of Deep Continuous Options for Robot Learning from DemonstrationsPreprint
Krishnan, S., Fox, R., Stoica, I. G., & Goldberg, K. Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.05421 · Cited by 53
Subgoal Discovery for Hierarchical Reinforcement Learning Using Learned Policies
Goel, S., & Huber, M. (2003) · The Florida AI Research Society · Cited by 50
The utility of temporal abstraction in reinforcement learning
Jong, N. K., Hester, T., & Stone, P. (2008) · doi:10.5555/1402383.1402429 · Cited by 45
Benchmarking Reinforcement Learning Algorithms on Real-World Robots
Mahmood, A. R., Korenkevych, D., Vasan, G., Ma, W., & Bergstra, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.07731 · Cited by 45
The Termination CriticPreprint
Harutyunyan, A., Dabney, W., Borsa, D., Heess, N., Munos, R., & Precup, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.09996 · Cited by 45
Scaling Up Approximate Value Iteration with Options: Better Policies with Fewer Iterations
Mann, T., & Mannor, S. (2014) · Cited by 44
Atrey, A., Clary, K., & Jensen, D. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.05743 · Cited by 44
Automated State Abstraction for Options using the U-Tree Algorithm
Jönsson, A., & Barto, A. G. (2000) · Cited by 43
Compositional Planning Using Optimal Option ModelsPreprint
Silver, D., & Ciosek, K. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6473 · Cited by 40
Learning complex motions by sequencing simpler motion templates
Neumann, G. M., Maass, W., & Peters, J. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553471 · Cited by 37
Finding and Visualizing Weaknesses of Deep Reinforcement Learning AgentsPreprint
Rupprecht, C., Ibrahim, C., & Pal, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01318 · Cited by 37
Research works printed but not linked: 53
- Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press.
- Li, Y. (2017). Deep Reinforcement Learning: An Overview. arXiv.
- Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv.
- Badia, A.P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, D., and Blundell, C. (2020). Agent57: Outperforming the Atari Human Benchmark. arXiv.
- OpenAI (2019). Dota 2 with Large Scale Deep Reinforcement Learning. arXiv.
- OpenAI (2019). Learning Dexterous In-Hand Manipulation. arXiv.
- Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., and Vanhoucke, V. (2018). QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. arXiv.
- Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete problems in AI safety. arXiv.
- Burda, Y., Edwards, H., Storkey, A., and Klimov, O. (2018). Exploration by Random Network Distillation. arXiv.
- Bellemare, M.G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. (2016). Unifying Count-Based Exploration and Intrinsic Motivation. arXiv.
- Nachum, O., Tang, H., Lu, X., Gu, S., Lee, H., and Levine, S. (2019). Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?. arXiv.
- Mott, A., Zoran, D., Chrzanowski, M., Wierstra, D., and Rezende, D.J. (2019). Towards Interpretable Reinforcement Learning Using Attention Augmented Agents. arXiv.
- Garnelo, M., Arulkumaran, K., and Shanahan, M. (2016). Towards Deep Symbolic Reinforcement Learning. arXiv.
- Houthooft, R., Chen, X., Chen, X., Duan, Y., Schulman, J., Turck, F.D., and Abbeel, P. (2016). VIME: Variational Information Maximizing Exploration. arXiv.
- Harutyunyan, A., Dabney, W., Mesnard, T., Heess, N., Azar, M.G., Piot, B., van Hasselt, H., Singh, S., Wayne, G., and Precup, D. (2019). Hindsight Credit Assignment. arXiv.
- Rummery, G.A., and Niranjan, M. (1994). On-Line Q-Learning Using Connectionist Systems, University of Cambridge, Department of Engineering.
- Donahue, J., and Simonyan, K. (2019). Large Scale Adversarial Representation Learning. arXiv.
- Radford, A., Metz, L., and Chintala, S. (2016). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv.
- van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016). WaveNet: A Generative Model for Raw Audio. arXiv.
- Sutskever, I., Vinyals, O., and Le, Q.V. (2014). Sequence to Sequence Learning with Neural Networks. arXiv.
- Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention Is All You Need. arXiv.
- Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative Adversarial Nets. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
- Van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. (2018). Deep Reinforcement Learning and the Deadly Triad. arXiv.
- Zhang, S., and Sutton, R.S. (2017). A Deeper Look at Experience Replay. arXiv.
- Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016). Prioritized Experience Replay. arXiv.
- Dabney, W., Rowland, M., Bellemare, M., and Munos, R. (2018, January 2–7). Distributional Reinforcement Learning with Quantile Regression. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA. · DOI · Paper page
- Ferns, N., Panangaden, P., and Precup, D. (2004, January 9–11). Metrics for Finite Markov Decision Processes. Proceedings of the Twentieth Conference on Uncertainty in Artificial Intelligence, Banff, AB, Canada.
- McGovern, A., Sutton, R.S., and Fagg, A.H. (1997, January 19–21). Roles of macro-actions in accelerating reinforcement learning. Proceedings of the Grace Hopper Celebration of Women in Computing, San Jose, CA, USA.
- Heess, N., Wayne, G., Tassa, Y., Lillicrap, T., Riedmiller, M., and Silver, D. (2016). Learning and transfer of modulated locomotor controllers. arXiv.
- Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. (2019). Diversity Is All You Need: Learning Skills without a Reward Function. arXiv.
- Abel, D., Umbanhowar, N., Khetarpal, K., Arumugam, D., Precup, D., and Littman, M.L. (2020, January 26–28). Value Preserving State-Action Abstractions. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, Palermo, Italy.
- Fox, R., Krishnan, S., Stoica, I., and Goldberg, K. (2017). Multi-level discovery of deep options. arXiv.
- Ng, A.Y., and Russell, S.J. (July, January 29). Algorithms for Inverse Reinforcement Learning. Proceedings of the International Conference on Machine Learning, Stanford, CA, USA.
- Pan, X., Ohn-Bar, E., Rhinehart, N., Xu, Y., Shen, Y., and Kitani, K.M. (2018). Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning. arXiv.
- Krishnan, S., Garg, A., Liaw, R., Miller, L., Pokorny, F.T., and Goldberg, K. (2016). Hirl: Hierarchical inverse reinforcement learning for long-horizon tasks with delayed rewards. arXiv.
- Kumar, A., Swersky, K., and Hinton, G. (2017, January 9). Feudal Learning for Large Discrete Action Spaces with Recursive Substructure. Proceedings of the NIPS Workshop Hierarchical Reinforcement Learning, Long Beach, CA, USA.
- Kulkarni, T.D., Saeedi, A., Gautam, S., and Gershman, S.J. (2016). Deep successor reinforcement learning. arXiv.
- Lakshminarayanan, A.S., Krishnamurthy, R., Kumar, P., and Ravindran, B. (2016). Option discovery in hierarchical reinforcement learning using spatio-temporal clustering. arXiv.
- Da Silva, B.C., Konidaris, G., and Barto, A.G. (July, January 26). Learning Parameterized Skills. Proceedings of the International Conference on Machine Learning, Edinburgh, Scotland.
- Wang, J.X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J.Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. (2017). Learning to Reinforcement Learn. arXiv.
- Duan, Y., Schulman, J., Chen, X., Bartlett, P.L., Sutskever, I., and Abbeel, P. (2016). RL$2^$: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv.
- Li, A.C., Florensa, C., Clavera, I., and Abbeel, P. (2020). Sub-Policy Adaptation for Hierarchical Reinforcement Learning. arXiv.
- Sohn, S., Woo, H., Choi, J., and Lee, H. (2020). Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies. arXiv.
- Bengio, E., Thomas, V., Pineau, J., Precup, D., and Bengio, Y. (2017). Independently Controllable Features. arXiv.
- Van Seijen, H., Fatemi, M., Romoff, J., Laroche, R., Barnes, T., and Tsang, J. (2017). Hybrid reward architecture for reinforcement learning. arXiv.
- Sukhbaatar, S., Denton, E., Szlam, A., and Fergus, R. (2018). Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning. arXiv.
- Gregor, K., Rezende, D.J., and Wierstra, D. (2016). Variational Intrinsic Control. arXiv.
- Achiam, J., Edwards, H., Amodei, D., and Abbeel, P. (2018). Variational option discovery algorithms. arXiv.
- Beattie, C., Leibo, J.Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., and Sadik, A. (2016). Deepmind lab. arXiv.
- Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets, A.S., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., and Schrittwieser, J. (2017). Starcraft ii: A new challenge for reinforcement learning. arXiv.
- Osband, I., Doron, Y., Hessel, M., Aslanides, J., Sezener, E., Saraiva, A., McKinney, K., Lattimore, T., Szepezvari, C., and Singh, S. (2019). Behaviour Suite for Reinforcement Learning. arXiv.
- Shang, W., Trott, A., Zheng, S., Xiong, C., and Socher, R. (2019). Learning World Graphs to Accelerate Hierarchical Reinforcement Learning. arXiv.
Cited by
Al-Hamadani, M. N. A., Fadhel, M. A., Alzubaidi, L., & Harangi, B. (2024) · Sensors · doi:10.3390/s24082461 · Cited by 123
A sensory–motor theory of the neocortex
Rao, R. P. N. (2024) · Nature Neuroscience · doi:10.1038/s41593-024-01673-9 · Cited by 103
Reinforcement learning for predictive maintenance: a systematic technical review
Siraskar, R., Kumar, S., Patil, S., Bongale, A. M., & Kotecha, K. V. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10468-6 · Cited by 92
Deep reinforcement learning challenges and opportunities for urban water systems
Negm, A. S., Ma, X., & Aggidis, G. (2024) · Water Research · doi:10.1016/j.watres.2024.121145 · Cited by 70
Reinforcement learning for vehicle-to-grid: A review
Xie, H., Song, G., Shi, Z., Zhang, J., Lin, Z., Yu, Q., Fu, H., Song, X., & Zhang, H. (2025) · Advances in Applied Energy · doi:10.1016/j.adapen.2025.100214 · Cited by 81
Metadata
- Authors
- Hutsebaut-Buysse, M., Mets, K., & Latré, S.
- Affiliations
- University of Antwerp (BE)
- Corresponding author
- Hutsebaut-Buysse, Matthias
- Venue
- Machine Learning and Knowledge Extraction
- Publisher
- MDPI AG
- Published
- 2022-02-17
- Volume
- 4
- Issue
- 1
- Pages
- 172-221
- ISSN
- 2504-4990 (ISSN-L)
- OpenAlex
- W4212996499
- Cited by
- 129
- References
- 208
- Citations by year
- 2026: 30 · 2025: 45 · 2024: 29 · 2023: 19 · 2022: 6
- FWCI
- 8.15
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Machine Learning and Knowledge Extraction (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- University of Antwerp (x)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Keywords
- hierarchical reinforcement learning, abstraction, temporal abstraction, state abstraction, options framework, goal-conditioned policies, sample efficiency, exploration, interpretability, high-dimensional environments, sequential decision making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.