The DOI matches its Crossref record; 327 of the 327 references it cites were found in a registry or an index.
Model-based Reinforcement Learning: A Survey
Open accessAbstract
Sequential decision making, commonly formalized as Markov Decision Process (MDP) optimization, is an important challenge in artificial intelligence. Two key approaches to this problem are reinforcement learning (RL) and planning. This survey is an integration of both fields, better known as model-based reinforcement learning. Model-based RL has two main steps. First, we systematically cover approaches to dynamics model learning, including challenges like dealing with stochasticity, uncertainty, partial observability, and temporal abstraction. Second, we present a systematic categorization of planning-learning integration, including aspects like: where to start planning, what budgets to allocate to planning and real data collection, how to plan, and how to integrate planning in the learning and acting loop. After these two sections, we also discuss implicit model-based RL as an end-to-end alternative for model learning and planning, and we cover the potential benefits of model-based RL. Along the way, the survey also draws connections to several related RL fields, like hierarchical RL and transfer learning. Altogether, the survey presents a broad conceptual overview of the combination of planning and learning for MDP optimization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Artificial intelligence: a modern approach
(1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 21,974
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
(2011) · Cited by 14,317
(2016) · Cited by 11,679
(1995) · Cited by 10,558
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
Identification and control of dynamical systems using neural networks
Narendra, K. S., & Parthasarathy, K. (1990) · IEEE Transactions on Neural Networks · doi:10.1109/72.80202 · Cited by 8,014
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
Caruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366
Söderström, T., & Stoica, P. (1988) · Cited by 5,162
Ljung, L. (1999) · Wiley Encyclopedia of Electrical and Electronics Engineering · doi:10.1002/047134608x.w1046 · Cited by 3,778
Rapidly-exploring random trees : a new tool for path planning
LaValle, S. M. (1998) · Cited by 3,704
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018) · Science · doi:10.1126/science.aar6404 · Cited by 3,562
An internal model for sensorimotor integration
Wolpert, D. M., Ghahramani, Z., & Jordan, M. I. (1995) · Science · doi:10.1126/science.7569931 · Cited by 3,553
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124
Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015) · arXiv · doi:10.48550/arxiv.1511.05952 · Cited by 3,120
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119
A Survey of Monte Carlo Tree Search Methods
Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., & Colton, S. (2012) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2012.2186810 · Cited by 2,970
Bandit Based Monte-Carlo Planning
Kocsis, L., & Szepesvári, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11871842_29 · Cited by 2,855
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
Pattern recognition and machine learning
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5091 · Cited by 2,650
Aleatory or epistemic? Does it matter?
Kiureghian, A. D., & Ditlevsen, O. (2009) · Structural Safety · doi:10.1016/j.strusafe.2008.06.020 · Cited by 2,454
Asymptotically efficient adaptive allocation rules
Lai, T. L., & Robbins, H. (1985) · Advances in Applied Mathematics · doi:10.1016/0196-8858(85)90002-8 · Cited by 2,406
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
Yu, L., Zhang, W., Wang, J., & Yu, Y. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10804 · Cited by 2,285
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161
Craik, K. J. W. (1943) · Cited by 1,845
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685
Self-improving reactive agents based on reinforcement learning, planning and teaching
Lin, L.-J. (1992) · Machine Learning · doi:10.1007/bf00992699 · Cited by 1,564
System identification—A survey
Åström, K. J., & Eykhoff, P. (1971) · Automatica · doi:10.1016/0005-1098(71)90059-8 · Cited by 1,495
Transfer Learning for Reinforcement Learning Domains: A Survey
Taylor, M. E., & Stone, P. (2009) · doi:10.5555/1577069.1755839 · Cited by 1,488
On-line Q-learning Using Connectionist Systems
Rummery, G. A. (1994) · Cited by 1,477
The theory of dynamic programming
Bellman, R. (1954) · Bulletin of the American Mathematical Society · doi:10.1090/s0002-9904-1954-09848-8 · Cited by 1,459
Forward models: supervised learning with a distal teacher
Jordan, M. I., & Rumelhart, D. E. (1992) · Cognitive Science · doi:10.1207/s15516709cog1603_1 · Cited by 1,451
Some studies in machine learning using the game of checkers
Samuel, A. L. (2000) · IBM Journal of Research and Development · doi:10.1147/rd.441.0206 · Cited by 1,439
Sutton, R. S. (1990) · Machine Learning Proceedings 1990 · doi:10.1016/b978-1-55860-141-3.50030-4 · Cited by 1,314
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314
Mastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · RePEc: Research Papers in Economics · Cited by 1,310
Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search
Coulom, R. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-75538-8_7 · Cited by 1,240
Using confidence bounds for exploitation-exploration trade-offs
Auer, P. L. (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944941 · Cited by 1,229
PILCO: A Model-Based and Data-Efficient Approach to Policy Search
Deisenroth, M. P., & Rasmussen, C. E. (2011) · Cambridge University Engineering Department Publications Database · Cited by 1,136
Intrinsic Motivation Systems for Autonomous Mental Development
Oudeyer, P.-Y., Kaplan, F., & Hafner, V. V. (2007) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2006.890271 · Cited by 1,126
Graves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058
Recent Advances in Hierarchical Reinforcement Learning
Barto, A. G., & Mahadevan, S. (2003) · Discrete Event Dynamic Systems · doi:10.1023/a:1022140919877 · Cited by 1,024
Predicting time series with support vector machines
Müller, K. R., Smola, A. J., Rätsch, G., Schölkopf, B., Kohlmorgen, J., & Vapnik, V. (1997) · Lecture Notes in Computer Science · doi:10.1007/bfb0020283 · Cited by 981
Interaction Networks for Learning about Objects, Relations and Physics
Battaglia, P. W., Pascanu, R., Lai, M., Rezende, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1612.00222 · Cited by 969
Korf, R. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90054-4 · Cited by 917
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Reinforcement Learning · doi:10.1007/978-1-4615-3618-5_2 · Cited by 903
Monte-Carlo Planning in Large POMDPs
Silver, D., & Veness, J. (2010) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 886
Transfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning
Nagabandi, A., Kahn, G., Fearing, R. S., & Levine, S. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8463189 · Cited by 848
Reinforcement learning for robots using neural networks
Lin, L.-J. (1992) · Defense Technical Information Center (DTIC) · Cited by 841
Prioritized sweeping: Reinforcement learning with less data and less time
Moore, A. W., & Atkeson, C. G. (1993) · Machine Learning · doi:10.1007/bf00993104 · Cited by 823
Near-optimal Regret Bounds for Reinforcement Learning
Jaksch, T., Ortner, R., & Auer, P. L. (2010) · Cited by 784
Dyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777
Learning Stable Nonlinear Dynamical Systems With Gaussian Mixture Models
Khansari-Zadeh, S. M., & Billard, A. (2011) · IEEE Transactions on Robotics · doi:10.1109/tro.2011.2159412 · Cited by 771
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 770
Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
Li, Z., Zhou, F., Chen, F., & Li, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.09835 · Cited by 763
SARSOP: Efficient Point-Based POMDP Planning by Approximating Optimally Reachable Belief Spaces
Kurniawati, H., Hsu, D., & Sun Lee, W. (2008) · Robotics: Science and Systems IV · doi:10.15607/rss.2008.iv.009 · Cited by 759
Unsupervised Learning for Physical Interaction through Video Prediction
Finn, C., Goodfellow, I., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1605.07157 · Cited by 759
(2016) · neural information processing systems · Cited by 735
Learning Latent Dynamics for Planning from Pixels
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. (2018) · arXiv · doi:10.48550/arxiv.1811.04551 · Cited by 708
Universal Value Function Approximators
Schaul, T., Horgan, D., Gregor, K., & Silver, D. (2015) · Cited by 701
Some Studies in Machine Learning Using the Game of Checkers. II—Recent Progress
Samuel, A. L. (1967) · IBM Journal of Research and Development · doi:10.1147/rd.116.0601 · Cited by 695
Improving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693
The Option-Critic Architecture
Bacon, P.-L., Harb, J., & Precup, D. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10916 · Cited by 687
Heuristic and analytic processes in reasoning*
Evans, J. S. B. T. (1984) · British Journal of Psychology · doi:10.1111/j.2044-8295.1984.tb01915.x · Cited by 622
Approximately Optimal Approximate Reinforcement Learning
Kakade, S. M., & Langford, J. C. (2002) · Cited by 617
Todorov, E., & Li, W. (2005) · Proceedings of the 2005, American Control Conference, 2005. · doi:10.1109/acc.2005.1469949 · Cited by 611
Intrinsically Motivated Reinforcement Learning
Singh, S., Barto, A. G., & Chentanez, N. (2005) · doi:10.21236/ada440280 · Cited by 610
Levine, S., & Koltun, V. (2013) · Cited by 586
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016) · arXiv · doi:10.48550/arxiv.1606.01540 · Cited by 582
Dream to Control: Learning Behaviors by Latent Imagination
Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2019) · arXiv · doi:10.48550/arxiv.1912.01603 · Cited by 558
Time-Contrastive Networks: Self-Supervised Learning from Video
Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., Levine, S., & Brain, G. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8462891 · Cited by 546
Curious model-building control systems
Schmidhuber, J. (1991) · [Proceedings] 1991 IEEE International Joint Conference on Neural Networks · doi:10.1109/ijcnn.1991.170605 · Cited by 541
Dayan, P., & Hinton, G. E. (1992) · Cited by 540
Provably safe and robust learning-based model predictive control
Aswani, A., Gonzalez, H., Sastry, S. S., & Tomlin, C. (2013) · Automatica · doi:10.1016/j.automatica.2013.02.003 · Cited by 539
Survey of Model-Based Reinforcement Learning: Applications on Robotics
Polydoros, A. S., & Nalpantidis, L. (2017) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-017-0468-y · Cited by 534
Locally Weighted Learning for Control
Atkeson, C. G., Moore, A. W., & Schaal, S. (1997) · Artificial Intelligence Review · doi:10.1023/a:1006511328852 · Cited by 527
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.12114 · Cited by 525
Model learning for robot control: A survey
Nguyen-Tuong, D., & Peters, J. (2011) · Cognitive Processing · doi:10.1007/s10339-011-0404-1 · Cited by 503
On the Sample Complexity of Reinforcement Learning
Kakade, S. M. (2003) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 502
Parameter estimation for linear dynamical systems
Ghahramani, Z., & Hinton, G. E. (1996) · Cambridge University Engineering Department Publications Database · Cited by 484
Tamar, A., Wu, Y., Thomas, G., Levine, S., & Abbeel, P. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/700 · Cited by 484
Diversity is All You Need: Learning Skills without a Reward Function
Eysenbach, B., Gupta, A., Ibarz, J., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1802.06070 · Cited by 480
Multiple model-based reinforcement learning
Doya, K., Samejima, K., Katagiri, K.-I., & Kawato, M. (2002) · Neural Computation · doi:10.1162/089976602753712972 · Cited by 476
VIME: Variational Information Maximizing Exploration
Houthooft, R., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09674 · Cited by 472
Active learning of inverse models with intrinsically motivated goal exploration in robots
Baranes, A., & Oudeyer, P.-Y. (2013) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2012.05.008 · Cited by 455
Stochastic Variational Video PredictionPreprint
Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R. H., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.11252 · Cited by 455
Dynamic Programming and Optimal Control. Volume 1: Dynamic Programming and Optimal Control. Volume 2
Smith, D. K. (1996) · Journal of the Operational Research Society · doi:10.1057/jors.1996.103 · Cited by 448
Data-efficient hierarchical reinforcement learning
Nachum, O., Gu, S., Lee, H., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.08296 · Cited by 446
Reinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446
Gaussian Process Dynamical Models
Wang, J., Hertzmann, A., & Blei, D. M. (2005) · Cited by 438
Deep Exploration via Bootstrapped DQN
Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04621 · Cited by 434
Recurrent World Models Facilitate Policy Evolution
Ha, D., & Schmidhuber, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.01999 · Cited by 429
Automatic Discovery of Subgoals in Reinforcement Learning using Diverse Density
McGovern, A., & Barto, A. G. (2001) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 420
Action-Conditional Video Prediction using Deep Networks in Atari Games
Oh, J., Guo, X., Lee, H., Lewis, R. L., & Singh, S. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.08750 · Cited by 419
Reinforcement learning with selective perception and hidden state
McCallum, A. K., & Ballard, D. H. (1996) · Cited by 411
Learning to Poke by Poking: Experiential Learning of Intuitive PhysicsPreprint
Agrawal, P., Nair, A., Abbeel, P., Malik, J., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.07419 · Cited by 404
The successor representation in human reinforcement learning
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., & Gershman, S. J. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0180-8 · Cited by 390
Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics
Levine, S., & Abbeel, P. (2014) · Cited by 389
The ubiquity of model-based reinforcement learning
Doll, B. B., Simon, D. A., & Daw, N. D. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.08.003 · Cited by 380
Count-Based Exploration with Neural Density Models
Ostrovski, G., Bellemare, M. G., Oord, A. V. D., & Munos, R. (2017) · arXiv · doi:10.48550/arxiv.1703.01310 · Cited by 377
Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning
Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., & Finn, C. (2018) · arXiv · doi:10.48550/arxiv.1803.11347 · Cited by 370
Lange, S., Gabel, T., & Riedmiller, M. (2012) · Adaptation, Learning, and Optimization · doi:10.1007/978-3-642-27645-3_2 · Cited by 370
Is Q-learning Provably Efficient?Preprint
Jin, C., Allen-Zhu, Z., Bubeck, S., & Jordan, M. I. (2018) · arXiv (Cornell University) · Cited by 368
R-MAX: a general polynomial time algorithm for near-optimal reinforcement learning
Brafman, R. I., & Tennenholtz, M. (2001) · Cited by 366
Dearden, R., Friedman, N., & Russell, S. (1998) · Cited by 363
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5699 · Cited by 359
A Compositional Object-Based Approach to Learning Physical DynamicsPreprint
Chang, M. B., Ullman, T. D., Torralba, A. B., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.00341 · Cited by 354
When to Trust Your Model: Model-Based Policy Optimization
Janner, M., Fu, J., Zhang, M., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1906.08253 · Cited by 351
Speed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346
Parameter Space Noise for Exploration
Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., & Andrychowicz, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.01905 · Cited by 346
Learning Continuous Control Policies by Stochastic Value Gradients
Heess, N., Wayne, G., Silver, D., Lillicrap, T., Tassa, Y., & Erez, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.09142 · Cited by 328
Hierarchical Recurrent Neural Networks for Long-Term Dependencies
Hihi, S. E., & Bengio, Y. (1995) · Cited by 325
Deep Reinforcement Learning that Matters
Henderson, P. B., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.06560 · Cited by 324
Teacher–Student Curriculum Learning
Matiisen, T., Oliver, A., Cohen, T., & Schulman, J. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2934906 · Cited by 320
Neural networks for control and system identification
Werbos, P. J. (2003) · Proceedings of the 28th IEEE Conference on Decision and Control · doi:10.1109/cdc.1989.70114 · Cited by 319
Reinforcement learning with perceptual aliasing: the perceptual distinctions approach
Chrisman, L. (1992) · Cited by 313
Successor Features for Transfer in Reinforcement Learning
Barreto, A. D. M. S., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., Silver, D., & van Hasselt, H. P. (2016) · arXiv (Cornell University) · doi:10.13140/rg.2.2.35439.18085 · Cited by 305
Continuous Deep Q-Learning with Model-based Acceleration
Gu, S., Lillicrap, T., Sutskever, I., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00748 · Cited by 304
State representation learning for control: An overview
Lesort, T., Díaz-Rodríguez, N., Goudou, J.-F., & Filliat, D. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.07.006 · Cited by 301
Safe Model-based Reinforcement Learning with Stability Guarantees
Berkenkamp, F., Turchetta, M., Schoellig, A. P., & Krause, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08551 · Cited by 299
A Deep Hierarchical Approach to Lifelong Learning in Minecraft
Tessler, C., Givony, S., Zahavy, T., Mankowitz, D., & Mannor, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10744 · Cited by 299
Robot learning from demonstration by constructing skill trees
Konidaris, G., Kuindersma, S., Grupen, R., & Barto, A. (2012) · The International Journal of Robotics Research · doi:10.1177/0278364911428653 · Cited by 286
Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
Stadie, B. C., Levine, S., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.00814 · Cited by 280
Autonomous helicopter control using reinforcement learning policy search methods
Bagnell, J. A., & Schneider, J. G. (2002) · Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164) · doi:10.1109/robot.2001.932842 · Cited by 276
Multi-Objective Decision Making
Roijers, D. M., & Whiteson, S. (2017) · Synthesis lectures on artificial intelligence and machine learning · doi:10.2200/s00765ed1v01y201704aim034 · Cited by 275
Reinforcement Learning with Soft State Aggregation
Singh, S. P., Jaakkola, T., & Jordan, M. I. (1994) · Cited by 274
Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning
Guo, X., Singh, S. P., Lee, H., Lewis, R. L., & Wang, X. (2014) · Cited by 271
Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning
Wu, J., Yildirim, I., Lim, J. J., Freeman, W. T., & Tenenbaum, J. B. (2016) · Cited by 264
Unifying Count-Based Exploration and Intrinsic Motivation
Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., & Munos, R. (2016) · arXiv · doi:10.48550/arxiv.1606.01868 · Cited by 264
Multi-task reinforcement learning: a hierarchical Bayesian approach
Wilson, A., Fern, A., Ray, S., & Tadepalli, P. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273624 · Cited by 261
Efficient Exploration in Reinforcement Learning
(2012) · SpringerReference · doi:10.1007/springerreference_178956 · Cited by 257
Transfer in Reinforcement Learning: A Framework and a Survey
Lazaric, A. (2012) · Adaptation, Learning, and Optimization · doi:10.1007/978-3-642-27645-3_5 · Cited by 256
End-to-End Differentiable Physics for Learning and Control
Belbute-Peres, F. D. A., Smith, K. A., Allen, K. R., Tenenbaum, J. B., & Kolter, J. Z. (2018) · neural information processing systems · Cited by 253
Reinforcement Learning with Unsupervised Auxiliary Tasks
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05397 · Cited by 247
Optimal learning: computational procedures for bayes-adaptive markov decision processes
Duff, M. O., & Barto, A. G. (2002) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 239
A comparison of direct and model-based reinforcement learning
Atkeson, C. G., & Santamaria, J. C. (2002) · Proceedings of International Conference on Robotics and Automation · doi:10.1109/robot.1997.606886 · Cited by 239
Identifying useful subgoals in reinforcement learning by local graph partitioning
Şimşek, Ö., Wolfe, A. P., & Barto, A. G. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102454 · Cited by 238
Bayesian Reinforcement Learning: A Survey
Ghavamzadeh, M., Mannor, S., Pineau, J., & Tamar, A. (2015) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000049 · Cited by 234
FeUdal Networks for Hierarchical Reinforcement Learning
Vezhnevets, A., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., & Kavukcuoglu, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01161 · Cited by 233
Learning Nonlinear Dynamical Systems Using an EM Algorithm
Ghahramani, Z., & Roweis, S. T. (1998) · Cited by 232
Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
Karl, M., Soelch, M., Bayer, J., & van der Smagt, P. (2016) · arXiv · doi:10.48550/arxiv.1605.06432 · Cited by 232
Robust Constrained Learning-based NMPC enabling reliable mobile robot path tracking
Ostafew, C. J., Schoellig, A. P., & Barfoot, T. D. (2016) · The International Journal of Robotics Research · doi:10.1177/0278364916645661 · Cited by 232
A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
Fraccaro, M., Kamronn, S., Paquet, U., & Winther, O. (2017) · arXiv · doi:10.48550/arxiv.1710.05741 · Cited by 229
Temporal Difference Models: Model-Free Deep RL for Model-Based ControlPreprint
Pong, V. H., Gu, S., Dalal, M., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09081 · Cited by 227
An object-oriented representation for efficient reinforcement learning
Diuk, C., Cohen, A., & Littman, M. L. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390187 · Cited by 226
Benchmarking Model-Based Reinforcement Learning
Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., Zhang, S., Zhang, G., Abbeel, P., & Ba, J. (2019) · arXiv · doi:10.48550/arxiv.1907.02057 · Cited by 226
(More) Efficient Reinforcement Learning via Posterior Sampling
Osband, I., Russo, D., & Roy, B. V. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1306.0940 · Cited by 222
Model-Based Bayesian Exploration
Dearden, R., Friedman, N., & André, D. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.6690 · Cited by 221
Automatic Goal Generation for Reinforcement Learning AgentsPreprint
Florensa, C., Held, D., Geng, X., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06366 · Cited by 221
Building portable options: skill transfer in reinforcement learning
Konidaris, G., & Barto, A. G. (2007) · Cited by 220
Q-Cut—Dynamic Discovery of Sub-goals in Reinforcement Learning
Menache, I., Mannor, S., & Shimkin, N. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-36755-1_25 · Cited by 218
Near-Bayesian exploration in polynomial time
Kolter, J. Z., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553441 · Cited by 218
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based ControlPreprint
Lowrey, K., Rajeswaran, A., Kakade, S. M., Todorov, E. V., & Mordatch, I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01848 · Cited by 213
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
Watter, M., Springenberg, J. T., Boedecker, J., & Riedmiller, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.07365 · Cited by 212
On-line policy improvement using Monte-Carlo search
Tesauro, G., & Galperin, G. R. (2025) · arXiv · doi:10.48550/arxiv.2501.05407 · Cited by 209
Near-optimal Regret Bounds for Reinforcement Learning
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859902 · Cited by 207
Krishnan, R. G., Shalit, U., & Sontag, D. (2015) · arXiv · doi:10.48550/arxiv.1511.05121 · Cited by 206
Go-Explore: a New Approach for Hard-Exploration Problems
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2019) · arXiv · doi:10.48550/arxiv.1901.10995 · Cited by 205
Imagination-Augmented Agents for Deep Reinforcement Learning
Weber, T., Racanière, S., Reichert, D., Buesing, L., Guez, A., Rezende, D. J., Badia, A. P., Vinyals, O., Heess, N., Li, Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06203 · Cited by 204
Markov Decision Processes: Markov Decision Processes: Discrete Stochastic Dynamic Programming
Thomas, L. C. (1995) · Journal of the Operational Research Society · doi:10.1057/jors.1995.110 · Cited by 204
SARSOP: Efficient Point-Based POMDP Planning by Approximating Optimally Reachable Belief Spaces
Kurniawati, H., Hsu, D., & Lee, W. S. (2009) · Robotics · doi:10.7551/mitpress/8344.003.0013 · Cited by 203
Gregor, K., Rezende, D. J., & Wierstra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07507 · Cited by 202
Learning an Embedding Space for Transferable Robot Skills
Hausman, K., Springenberg, J. T., Wang, Z., Heess, N., & Riedmiller, M. (2018) · International Conference on Learning Representations · Cited by 202
Data-Efficient Reinforcement Learning with Probabilistic Model Predictive ControlPreprint
Kamthe, S., & Deisenroth, M. P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06491 · Cited by 199
Thinking Fast and Slow with Deep Learning and Tree Search
Anthony, T., Tian, Z., & Barber, D. (2017) · arXiv · doi:10.48550/arxiv.1705.08439 · Cited by 198
Improving Multi-Step Prediction of Learned Time Series Models
Venkatraman, A., Hebert, M., & Bagnell, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9590 · Cited by 198
Learning and Transfer of Modulated Locomotor ControllersPreprint
Heess, N., Wayne, G., Tassa, Y., Lillicrap, T., Riedmiller, M., & Silver, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.05182 · Cited by 198
Action-conditional video prediction using deep networks in Atari games
Oh, J., Guo, X., Lee, H., Lewis, R. L., & Singh, S. P. (2015) · Cited by 196
The Predictron: End-To-End Learning and PlanningPreprint
Silver, D., van Hasselt, H. P., Hessel, M., Schaul, T., Guez, A., Harley, T., Dulac-Arnold, G., Reichert, D., Rabinowitz, N. C., Barreto, A. D. M. S., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08810 · Cited by 195
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their InteractionsPreprint
van Steenkiste, S., Chang, M., Greff, K., & Schmidhuber, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10353 · Cited by 192
Dynamic abstraction in reinforcement learning via clustering
Mannor, S., Menache, I., Hoze, A., & Klein, U. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015355 · Cited by 191
Variational Option Discovery AlgorithmsPreprint
Achiam, J., Edwards, H. S., Amodei, D., & Abbeel, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.10299 · Cited by 188
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2012) · arXiv · doi:10.48550/arxiv.1207.4708 · Cited by 187
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLPreprint
Nagabandi, A., Finn, C., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.07671 · Cited by 184
Contrastive Learning of Structured World ModelsPreprint
Kipf, T., van der Pol, E., & Welling, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.12247 · Cited by 184
Finding Structure in Reinforcement Learning
Thrun, S., & Schwartz, A. (1994) · Cited by 183
A Differentiable Physics Engine for Deep Learning in Robotics
Degrave, J., Hermans, M., Dambre, J., & wyffels, F. (2019) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2019.00006 · Cited by 182
Stochastic Neural Networks for Hierarchical Reinforcement Learning
Florensa, C., Duan, Y., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03012 · Cited by 180
Error bounds for approximate policy iteration
Munos, R. (2003) · Cited by 179
SOLAR: Deep Structured Representations for Model-Based Reinforcement LearningPreprint
Zhang, M., Vikram, S., Smith, L., Abbeel, P., Johnson, M. J., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.09105 · Cited by 176
Learning Visual Predictive Models of Physics for Playing BilliardsPreprint
Fragkiadaki, K., Agrawal, P., Levine, S., & Malik, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.07404 · Cited by 172
Parr, R., Li, L. C., Taylor, G., Painter-Wakefield, C., & Littman, M. L. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390251 · Cited by 164
Blundell, C., Uria, B., Pritzel, A., Li, Y., Ruderman, A., Leibo, J. Z., Rae, J., Wierstra, D., & Hassabis, D. (2016) · arXiv · doi:10.48550/arxiv.1606.04460 · Cited by 162
Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
Buckman, J., Hafner, D., Tucker, G., Brevdo, E., & Lee, H. (2018) · arXiv · doi:10.48550/arxiv.1807.01675 · Cited by 161
Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
Kansky, K., Silver, T., Mély, D. A., Eldawy, M., Lázaro-Gredilla, M., Lou, X., Dorfman, N., Sidor, S., Phoenix, S., & George, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.04317 · Cited by 161
Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching
Lin, L.-J. (1992) · Reinforcement Learning · doi:10.1007/978-1-4615-3618-5_5 · Cited by 160
Transfer from Simulation to Real World through Learning Deep Inverse Dynamics ModelPreprint
Christiano, P. F., Shah, Z., Mordatch, I., Schneider, J., Blackwell, T., Tobin, J., Abbeel, P., & Zaremba, W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.03518 · Cited by 159
Cited by
A review of cooperative multi-agent deep reinforcement learning
Oroojlooy, A., & Hajinezhad, D. (2023) · Applied Intelligence · doi:10.1007/s10489-022-04105-y · Cited by 482
Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods
Cao, Y., Zhao, H., Cheng, Y., Shu, T., Chen, Y., Liu, G., Liang, G., Zhao, J., Yan, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3497992 · Cited by 174
World Models for Autonomous Driving: An Initial Survey
Guan, Y., Liao, H., Li, Z., Hu, J., Yuan, R., Zhang, G., & Xu, C. (2025) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3398357 · Cited by 76
Agentic Large Language Models, a Survey
Plaat, A., Van Duijn, M., Van Stein, N., Preuss, M., Van der Putten, P., & Batenburg, K. J. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.18675 · Cited by 75
Self-citationUser Simulation for Evaluating Information Access Systems
Balog, K., & Zhai, C. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000098 · Cited by 52
Metadata
- Authors
- M. Moerland, T., Broekens, J., Plaat, A., & M. Jonker, C.
- Venue
- Foundations and Trends in Machine Learning
- Publisher
- Emerald
- Published
- 2023-01-04
- Volume
- 16
- Issue
- 1
- Pages
- 1-118
- ISSN
- 1935-8237 (ISSN-L)
- 1935-8245
- OpenAlex
- W3038822267
- Cited by
- 524
- References
- 327
- FWCI
- 39.80
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.