The DOI matches its Crossref record; 327 of the 327 references it cites were found in a registry or an index.
Model-based Reinforcement Learning: A Survey
Open accessAbstract
Sequential decision making, commonly formalized as Markov Decision Process (MDP) optimization, is an important challenge in artificial intelligence. Two key approaches to this problem are reinforcement learning (RL) and planning. This survey is an integration of both fields, better known as model-based reinforcement learning. Model-based RL has two main steps. First, we systematically cover approaches to dynamics model learning, including challenges like dealing with stochasticity, uncertainty, partial observability, and temporal abstraction. Second, we present a systematic categorization of planning-learning integration, including aspects like: where to start planning, what budgets to allocate to planning and real data collection, how to plan, and how to integrate planning in the learning and acting loop. After these two sections, we also discuss implicit model-based RL as an end-to-end alternative for model learning and planning, and we cover the potential benefits of model-based RL. Along the way, the survey also draws connections to several related RL fields, like hierarchical RL and transfer learning. Altogether, the survey presents a broad conceptual overview of the combination of planning and learning for MDP optimization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–327 of 327
Learning state representations with robotic priors
Jonschkowski, R., & Brock, O. (2015) · Autonomous Robots · doi:10.1007/s10514-015-9459-7 · Cited by 158
Generalizing plans to new environments in relational MDPs
Guestrin, C., Koller, D., Gearhart, C., & Kanodia, N. (2003) · Cited by 157
A Bayesian Sampling Approach to Exploration in Reinforcement LearningPreprint
Asmuth, J., Li, L. C., Littman, M. L., Nouri, A., & Wingate, D. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.2664 · Cited by 156
Recurrent Environment SimulatorsPreprint
Chiappa, S., Racanière, S., Wierstra, D., & Mohamed, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.02254 · Cited by 156
Dealing with non-stationary environments using context detection
da Silva, B. C., Basso, E. W., Bazzan, A. L. C., & Engel, P. M. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143872 · Cited by 154
Stable reinforcement learning with autoencoders for tactile and visual data
van Hoof, H., Chen, N., Karl, M., van der Smagt, P., & Peters, J. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759578 · Cited by 149
Locally Weighted Learning for Control
Atkeson, C. G., Moore, A., & Schaal, S. (1997) · Springer eBooks · doi:10.1007/978-94-017-2053-3_3 · Cited by 149
Model-Based Reinforcement Learning via Meta-Policy OptimizationPreprint
Clavera, I., Rothfuss, J., Schulman, J., Fujita, Y., Asfour, T., & Abbeel, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.05214 · Cited by 148
On-line regression algorithms for learning mechanical models of robots: A survey
Sigaud, O., Salaün, C., & Padois, V. (2011) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2011.07.006 · Cited by 144
Imagination-Augmented Agents for Deep Reinforcement LearningPreprint
Racanière, S., Weber, T., Reichert, D., Buesing, L., Guez, A., Rezende, D. J., Badia, A. P., Vinyals, O., Heess, N., Li, Y., et al. (2017) · arXiv (Cornell University) · Cited by 144
R-IAC: Robust Intrinsically Motivated Exploration and Active Learning
Baranès, A., & Oudeyer, P. (2009) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2009.2037513 · Cited by 142
Why is Posterior Sampling Better than Optimism for Reinforcement Learning?Preprint
Osband, I., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.00215 · Cited by 141
Exploration in Model-based Reinforcement Learning by Empirically Estimating Learning Progress
Lopes, M. C., Lang, T., Toussaint, M., & Oudeyer, P. (2012) · Cited by 140
Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
Achiam, J., & Sastry, S. (2017) · arXiv · doi:10.48550/arxiv.1703.01732 · Cited by 139
Dyna-Style Planning with Linear Function Approximation and Prioritized SweepingPreprint
Sutton, R. S., Szepesvári, C., Geramifard, A., & Bowling, M. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.3285 · Cited by 139
Solving the Rubik’s cube with deep reinforcement learning and search
Agostinelli, F., McAleer, S., Shmakov, A., & Baldi, P. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0070-z · Cited by 136
From Pixels to Torques: Policy Learning with Deep Dynamical ModelsPreprint
Wahlström, N., Schön, T. B., & Deisenroth, M. P. (2015) · arXiv (Cornell University) · doi:10.25561/21120 · Cited by 134
A point-based POMDP algorithm for robot planning
Spaan, M. T. J., & Spaan, N. (2004) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2004.1307420 · Cited by 130
Planning to Explore via Self-Supervised World Models
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., & Pathak, D. (2020) · arXiv · doi:10.48550/arxiv.2005.05960 · Cited by 129
Behaviour Suite for Reinforcement LearningPreprint
Osband, I., Doron, Y., Hessel, M., Aslanides, J., Sezener, E., Saraiva, A., McKinney, K., Lattimore, T., Szepesvári, C., Singh, S. P., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.03568 · Cited by 128
Automatic Goal Generation for Reinforcement Learning Agents
Florensa, C., Held, D., Geng, X., & Abbeel, P. (2018) · International Conference on Machine Learning · Cited by 127
One-shot learning of manipulation skills with online dynamics adaptation and neural network priors
Fu, J., Levine, S., & Abbeel, P. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759592 · Cited by 126
Minimax Regret Bounds for Reinforcement Learning
Azar, M. G., Osband, I., & Munos, R. (2017) · arXiv · doi:10.48550/arxiv.1703.05449 · Cited by 124
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning
Eysenbach, B., Salakhutdinov, R., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1906.05253 · Cited by 123
Sample Complexity of Episodic Fixed-Horizon Reinforcement LearningPreprint
Dann, C., & Brunskill, E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.08906 · Cited by 122
Learning Plannable Representations with Causal InfoGANPreprint
Kurutach, T., Tamar, A., Yang, G., Russell, S., & Abbeel, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.09341 · Cited by 121
A Laplacian Framework for Option Discovery in Reinforcement LearningPreprint
Machado, M. C., Bellemare, M. G., & Bowling, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00956 · Cited by 120
Watters, N., Matthey, L., Bošnjak, M., Burgess, C., & Lerchner, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09275 · Cited by 119
Learning and Querying Fast Generative Models for Reinforcement LearningPreprint
Buesing, L., Weber, T., Racanière, S., Eslami, S. M. A., Rezende, D. J., Reichert, D., Viola, F., Besse, F., Gregor, K., Hassabis, D., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03006 · Cited by 118
Giraffe: Using Deep Reinforcement Learning to Play ChessPreprint
Lai, M. (2015) · arXiv.org · doi:10.48550/arxiv.1509.01549 · Cited by 115
Learning model-based planning from scratchPreprint
Pascanu, R., Li, Y., Vinyals, O., Heess, N., Buesing, L., Racanière, S., Reichert, D., Weber, T., Wierstra, D., & Battaglia, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06170 · Cited by 113
When to use parametric models in reinforcement learning?Preprint
van Hasselt, H. P., Hessel, M., & Aslanides, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05243 · Cited by 113
TEXPLORE: real-time sample-efficient reinforcement learning for robots
Hester, T., & Stone, P. (2012) · Machine Learning · doi:10.1007/s10994-012-5322-7 · Cited by 112
Multi-Level Discovery of Deep OptionsPreprint
Fox, R., Krishnan, S., Stoica, I. G., & Goldberg, K. Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.08294 · Cited by 111
Sample-based learning and search with permanent and transient memories
Silver, D., Sutton, R. S., & Müller, M. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390278 · Cited by 109
Unsupervised State Representation Learning in AtariPreprint
Anand, A., Racah, E., Ozair, S., Bengio, Y., Côté, M.-A., & Hjelm, R. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.08226 · Cited by 109
Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based SearchPreprint
Guez, A., Silver, D., & Dayan, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.3109 · Cited by 107
Model-based reinforcement learning with nearly tight exploration complexity bounds
Szita, I., & ri, C. S. (2010) · Cited by 106
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement LearningPreprint
Shu, T., Xiong, C., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.07294 · Cited by 106
Contrastive Explanations for Reinforcement Learning in terms of Expected ConsequencesPreprint
van der Waa, J., van Diggelen, J., den Bosch, K. V., & Neerincx, M. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.08706 · Cited by 103
Model-Based Active ExplorationPreprint
Shyam, P., Jaśkowski, W., & Gomez, F. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12162 · Cited by 102
Learning Actionable Representations with Goal-Conditioned PoliciesPreprint
Ghosh, D., Gupta, A., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.07819 · Cited by 101
Memory Approaches to Reinforcement Learning in Non-Markovian Domains
Lin, L., & Mitchell, T. M. (1992) · Cited by 100
Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
Dilokthanakul, N., Kaplanis, C., Pawlowski, N., & Shanahan, M. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2891792 · Cited by 100
Probabilistic inference for determining options in reinforcement learning
Daniel, C., van Hoof, H., Peters, J., & Neumann, G. M. (2016) · Machine Learning · doi:10.1007/s10994-016-5580-x · Cited by 95
Uncertainty-driven Imagination for Continuous Deep Reinforcement Learning
Kalweit, G., & Boedecker, J. (2017) · Conference on Robot Learning · Cited by 95
Learning Policies with External MemoryPreprint
Peshkin, L., Meuleau, N., & Kaelbling, L. P. (2001) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0103003 · Cited by 92
Learning to act using real-time dynamic programming
Barto, A. G., Bradtke, S. J., & Singh, S. P. (1996) · The MIT Press eBooks · doi:10.7551/mitpress/2026.003.0005 · Cited by 92
Episodic Memory Deep Q-Networks
Lin, Z., Zhao, T., Yang, G., & Zhang, L. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/337 · Cited by 92
Some studies in machine learning using the game of checkers. II—Recent progress
Samuel, A. L. (1969) · Annual Review in Automatic Programming · doi:10.1016/0066-4138(69)90004-4 · Cited by 89
Analogues of mental simulation and imagination in deep learning
Hamrick, J. B. (2019) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2018.12.011 · Cited by 89
Latent Normalizing Flows for Discrete SequencesPreprint
Ziegler, Z. M., & Rush, A. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10548 · Cited by 88
Zanette, A., & Brunskill, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.00210 · Cited by 86
How can we define intrinsic motivation
Oudeyer, P., & Kaplan, F. (2008) · Cited by 84
TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement LearningPreprint
Farquhar, G., Rocktäschel, T., Igl, M., & Whiteson, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.11417 · Cited by 82
Combined Reinforcement Learning via Abstract Representations
François-Lavet, V., Bengio, Y., Precup, D., & Pineau, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013582 · Cited by 82
PAC-inspired Option Discovery in Lifelong Reinforcement Learning
Brunskill, E., & Li, L. (2014) · Cited by 79
Generative Temporal Models with MemoryPreprint
Gemici, M., Hung, C.-C., Santoro, A., Wayne, G., Mohamed, S., Rezende, D. J., Amos, D., & Lillicrap, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.04649 · Cited by 78
Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks
Depeweg, S., Hernández-Lobato, J. M., Doshi‐Velez, F., & Udluft, S. (2016) · Cambridge University Engineering Department Publications Database · doi:10.17863/cam.55902 · Cited by 75
Curiosity Driven Exploration of Learned Disentangled Goal SpacesPreprint
Laversanne-Finot, A., Péré, A., & Oudeyer, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.01521 · Cited by 75
Recent Advances in Hierarchical Reinforcement Learning
Barto, A. G., & Mahadevan, S. (2003) · Cited by 74
Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal ExplorationPreprint
Péré, A., Forestier, S., Sigaud, O., & Oudeyer, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.00781 · Cited by 70
Contingency-Aware Exploration in Reinforcement LearningPreprint
Choi, J., Guo, Y., Moczulski, M., Oh, J., Wu, N., Norouzi, M., & Lee, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01483 · Cited by 70
Thompson Sampling for Learning Parameterized Markov Decision ProcessesPreprint
Gopalan, A., & Mannor, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.7498 · Cited by 69
Hierarchical Reinforcement Learning with HindsightPreprint
Levy, A., Platt, R., & Saenko, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08180 · Cited by 66
Efficient Model-Based Exploration
Wiering, M., & Schmidhuber, J. (1998) · The MIT Press eBooks · doi:10.7551/mitpress/3119.003.0034 · Cited by 63
Contrastive Learning of Structured World Models
Kipf, T., van der Pol, E., & Welling, M. (2020) · Data Archiving and Networked Services (DANS) · Cited by 63
Disentangling the independently controllable factors of variation by interacting with the worldPreprint
Thomas, V., Bengio, E., Fedus, W., Pondard, J., Beaudoin, P., Larochelle, H., Pineau, J., Precup, D., & Bengio, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09484 · Cited by 61
Model-based function approximation in reinforcement learning
Jong, N. K., & Stone, P. (2007) · Adaptive Agents and Multi-Agents Systems · doi:10.1145/1329125.1329242 · Cited by 60
Unsupervised Discovery of Parts, Structure, and DynamicsPreprint
Xu, Z., Liu, Z., Sun, C., Murphy, K. P., Freeman, W. T., Tenenbaum, J. B., & Wu, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.05136 · Cited by 60
Fröhlich, F., Theis, F. J., & Hasenauer, J. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-12982-2_5 · Cited by 57
Learning Representation and Control in Markov Decision Processes: New Frontiers
Sridhar, M. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000003 · Cited by 55
Metacontrol for Adaptive Imagination-Based OptimizationPreprint
Hamrick, J. B., Ballard, A. J., Pascanu, R., Vinyals, O., Heess, N., & Battaglia, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.02670 · Cited by 54
Learning to Search with MCTSnetsPreprint
Guez, A., Weber, T., Antonoglou, I., Simonyan, K., Vinyals, O., Wierstra, D., Munos, R., & Silver, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04697 · Cited by 54
Efficient Exploration in Reinforcement Learning
Langford, J. C. (2017) · Springer eBooks · doi:10.1007/978-1-4899-7687-1_244 · Cited by 53
Efficient Model-Based Deep Reinforcement Learning with Variational State TabulationPreprint
Corneil, D., Wulfram Gerstner, B., & Brea, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04325 · Cited by 53
Time-Agnostic Prediction: Predicting Predictable Video FramesPreprint
Jayaraman, D., Ebert, F., Efros, A. A., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07784 · Cited by 53
(2009) · Cited by 52
Efficient reinforcement learning: model-based Acrobot control
Boone, G. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1997.620043 · Cited by 51
Model regularization for stable sample rollouts
Talvitie, E. (2014) · Cited by 51
Subgoal Discovery for Hierarchical Reinforcement Learning Using Learned Policies
Goel, S., & Huber, M. (2003) · The Florida AI Research Society · Cited by 50
Learning Abstract OptionsPreprint
Riemer, M., Liu, M., & Tesauro, G. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.11583 · Cited by 50
Fairbank, M., & Alonso, E. (2012) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2012.6252791 · Cited by 47
Option Discovery in Hierarchical Reinforcement Learning using Spatio-Temporal ClusteringPreprint
Srinivas, A., Krishnamurthy, R., Kumar, P., & Ravindran, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.05359 · Cited by 47
Forward-Backward Reinforcement LearningPreprint
Edwards, A. D., Downs, L., & Davidson, J. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.10227 · Cited by 47
Model-based multi-objective reinforcement learning
Wiering, M., Withagen, M., & Drugan, M. M. (2014) · IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning · doi:10.1109/adprl.2014.7010622 · Cited by 45
Scaling Up Approximate Value Iteration with Options: Better Policies with Fewer Iterations
Mann, T., & Mannor, S. (2014) · Cited by 44
On Lower Bounds for Regret in Reinforcement LearningPreprint
Osband, I., & Roy, B. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.02732 · Cited by 44
Self-Correcting Models for Model-Based Reinforcement Learning
Talvitie, E. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10850 · Cited by 42
Efficient Model-Based Exploration
Wiering, M., & Schmidhuber, J. (1998) · Utrecht University Repository (Utrecht University) · Cited by 42
Hester, T., & Stone, P. (2012) · Adaptation, learning, and optimization · doi:10.1007/978-3-642-27645-3_4 · Cited by 41
Combining Q-Learning and Search with Amortized Value EstimatesPreprint
Hamrick, J. B., Bapst, V., Sánchez‐González, Á., Pfaff, T., Weber, T., Buesing, L., & Battaglia, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02807 · Cited by 41
A Deeper Look at Planning as Learning from Replay
van Seijen, H., & Sutton, R. (2015) · International Conference on Machine Learning · Cited by 40
Efficient exploration with Double Uncertain Value NetworksPreprint
Moerland, T. M., Broekens, J., & Jonker, C. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10789 · Cited by 39
Self-citationLearning Multimodal Transition Dynamics for Model-Based Reinforcement LearningPreprint
Moerland, T. M., Broekens, J., & Jonker, C. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.00470 · Cited by 38
Self-citationGeneralizable Episodic Memory for Deep Reinforcement LearningPreprint
Hu, H., Ye, J., Zhu, G., Ren, Z., & Zhang, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06469 · Cited by 36
Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term FuturePreprint
Ke, N. R., Singh, A., Touati, A., Goyal, A., Bengio, Y., Parikh, D., & Batra, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.01599 · Cited by 35
A0C: Alpha Zero in Continuous Action Space
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2018) · arXiv · doi:10.48550/arxiv.1805.09613 · Cited by 34
Self-citationIntrinsically motivated model learning for a developing curious agent
Hester, T., & Stone, P. (2012) · doi:10.1109/devlrn.2012.6400802 · Cited by 33
TDLeaf(lambda): Combining Temporal Difference Learning with Game-Tree SearchPreprint
Baxter, J., Tridgell, A., & Weaver, L. (1999) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/9901001 · Cited by 32
Learning by appraising: an emotion-based approach to intrinsic reward design
Sequeira, P., Melo, F. S., & Paiva, A. (2014) · Adaptive Behavior · doi:10.1177/1059712314543837 · Cited by 32
World Model as a Graph: Learning Latent Landmarks for PlanningPreprint
Zhang, L., Yang, G., & Stadie, B. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.12491 · Cited by 32
A Short Survey On Memory Based Reinforcement LearningPreprint
Ramani, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.06736 · Cited by 31
The Effect of Planning Shape on Dyna-style Planning in High-dimensional State SpacesPreprint
Holland, G. Z., Talvitie, E. J., & Bowling, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.01825 · Cited by 31
Variance Reduction Methods for Sublinear Reinforcement LearningPreprint
Kakade, S. M., Wang, M., & Yang, L. F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09184 · Cited by 30
Policy Gradient Search: Online Planning and Expert Iteration without Search TreesPreprint
Anthony, T., Nishihara, R., Moritz, P., Salimans, T., & Schulman, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03646 · Cited by 29
Beyond the One Step Greedy Approach in Reinforcement LearningPreprint
Efroni, Y., Dalal, G., Scherrer, B., & Mannor, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03654 · Cited by 26
Learning first-order Markov models for control
Abbeel, P., & Ng, A. Y. (2004) · Cited by 25
Tight Regret Bounds for Model-Based Reinforcement Learning with Greedy PoliciesPreprint
Efroni, Y., Merlis, N., Ghavamzadeh, M., & Mannor, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11527 · Cited by 24
A Framework for Reinforcement Learning and Planning.Preprint
Moerland, T. M., Broekens, J., & Jonker, C. M. (2020) · arXiv (Cornell University) · Cited by 20
Self-citationThe Value Equivalence Principle for Model-Based Reinforcement LearningPreprint
Grimm, C., Barreto, A. D. M. S., Singh, S. P., & Silver, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.03506 · Cited by 19
Posterior sampling for reinforcement learning: worst-case regret boundsPreprint
Agrawal, S., & Jia, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07041 · Cited by 17
Self-Correcting Models for Model-Based Reinforcement LearningPreprint
Talvitie, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.06018 · Cited by 17
Adaptive Online Planning for Continual Lifelong LearningPreprint
Lu, K., Mordatch, I., & Abbeel, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.01188 · Cited by 15
Towards a Simple Approach to Multi-step Model-based Reinforcement LearningPreprint
Asadi, K., Cater, E., Misra, D. K., & Littman, M. L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00128 · Cited by 13
Model-Based Deep Reinforcement Learning for High-Dimensional Problems, a Survey.Preprint
Plaat, A., Kosters, W. A., & Preuß, M. (2020) · arXiv (Cornell University) · Cited by 12
Self-citationAdaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical ModelsPreprint
Neitz, A., Parascandolo, G., Bauer, S., & Schölkopf, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.04768 · Cited by 10
AMRL: Aggregated Memory For Reinforcement Learning
Beck, J., Ciosek, K., Devlin, S., Tschiatschek, S., Zhang, C., & Hofmann, K. (2020) · International Conference on Learning Representations · Cited by 10
Disentangling Controllable and Uncontrollable Factors of Variation by Interacting with the WorldPreprint
Sawada, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.06955 · Cited by 9
Model-Based Multi-objective Reinforcement Learning with Unknown Weights
Yamaguchi, T., Nagahama, S., Ichikawa, Y., & Takadama, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-22649-7_25 · Cited by 9
Value targets in off-policy AlphaZero: a new greedy backup
Willemsen, D., Baier, H., & Kaisers, M. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-05928-5 · Cited by 9
Think Too Fast Nor Too Slow: The Computational Trade-off Between Planning And Reinforcement LearningPreprint
Moerland, T. M., Deichler, A., Baldi, S., Broekens, J., & Jonker, C. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07404 · Cited by 7
Self-citationThe LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement LearningPreprint
van Seijen, H., Nekoei, H., Racah, E., & Chandar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03158 · Cited by 4
Now I Remember! Episodic Memory For Reinforcement Learning
Loynd, R., Hausknecht, M., Li, L., & Li Juan, D. E. N. G. (2018) · Cited by 1
Pritzel, A., Uria, B., Srinivasan, S., Puigdomènech, A., Vinyals, O., Hassabis, D., Wierstra, D., & Blundell, C. (2017) · arXiv · doi:10.48550/arxiv.1703.01988
Generalization of Reinforcement Learners with Working and Episodic Memory
Fortunato, M., Tan, M., Faulkner, R., Hansen, S., Badia, A. P., Buttimore, G., Deck, C., Leibo, J. Z., & Blundell, C. (2019) · arXiv · doi:10.48550/arxiv.1910.13406
Cited by
A review of cooperative multi-agent deep reinforcement learning
Oroojlooy, A., & Hajinezhad, D. (2023) · Applied Intelligence · doi:10.1007/s10489-022-04105-y · Cited by 482
Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods
Cao, Y., Zhao, H., Cheng, Y., Shu, T., Chen, Y., Liu, G., Liang, G., Zhao, J., Yan, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3497992 · Cited by 174
World Models for Autonomous Driving: An Initial Survey
Guan, Y., Liao, H., Li, Z., Hu, J., Yuan, R., Zhang, G., & Xu, C. (2025) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3398357 · Cited by 76
Agentic Large Language Models, a Survey
Plaat, A., Van Duijn, M., Van Stein, N., Preuss, M., Van der Putten, P., & Batenburg, K. J. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.18675 · Cited by 75
Self-citationUser Simulation for Evaluating Information Access Systems
Balog, K., & Zhai, C. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000098 · Cited by 52
Metadata
- Authors
- M. Moerland, T., Broekens, J., Plaat, A., & M. Jonker, C.
- Venue
- Foundations and Trends in Machine Learning
- Publisher
- Emerald
- Published
- 2023-01-04
- Volume
- 16
- Issue
- 1
- Pages
- 1-118
- ISSN
- 1935-8237 (ISSN-L)
- 1935-8245
- OpenAlex
- W3038822267
- Cited by
- 524
- References
- 327
- FWCI
- 39.80
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.