A survey on model-based reinforcement learning
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Reinforcement learning (RL) interacts with the environment to solve sequential decision-making problems via a trial-and-error approach. Errors are always undesirable in real-world applications, even though RL excels at playing complex video games that permit several trial-and-error attempts. To improve sample efficiency and thus reduce errors, model-based reinforcement learning (MBRL) is believed to be a promising direction, as it constructs environment models in which trial-and-errors can occur without incurring actual costs. In this survey, we investigate MBRL with a particular focus on the recent advancements in deep RL. There is a generalization error between the learned model of a non-tabular environment and the actual environment. Consequently, it is crucial to analyze the disparity between policy training in the environment model and that in the actual environment, guiding algorithm design for improved model learning, model utilization, and policy training. In addition, we discuss the recent developments of model-based techniques in other forms of RL, such as offline RL, goal-conditioned RL, multi-agent RL, and meta-RL. Furthermore, we discuss the applicability and benefits of MBRL for real-world tasks. Finally, this survey concludes with a discussion of the promising future development prospects for MBRL. We believe that MBRL has great unrealized potential and benefits in real-world applications, and we hope this survey will encourage additional research on MBRL.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084 · cited 3×
“AlphaGo Zero [Silver et al., 2017b] is able to defeat professional human players without any human knowledge.”
BackgroundAsynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · cited 1×
“Moreover, MBPO also adopts Soft Actor-Critic [Haarnoja et al., 2018], which is an off-policy RL algorithm, to update the policy with the mixed data from the real environment and learned model.”
BackgroundTrust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119
A Survey of Monte Carlo Tree Search Methods
Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., & Colton, S. (2012) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2012.2186810 · Cited by 2,970 · cited 2×
“At each time step, MCTS incrementally extends a search tree from the current environment state [Browne et al., 2012, Chaslot et al., 2008a].”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Temporal difference learning and TD-Gammon
Tesauro, G. (1995) · Communications of the ACM · doi:10.1145/203330.203343 · Cited by 1,618 · cited 1×
“Another value approximation method is temporal-difference (TD) prediction [Tesauro, 1995], which is broadly used in many value-based RL methods [Lillicrap et al., 2016, Mnih et al., 2015].”
MethodLearning agile and dynamic motor skills for legged robots
Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V., & Hutter, M. (2019) · Science Robotics · doi:10.1126/scirobotics.aau5872 · Cited by 1,482 · cited 1×
“Moreover, Hwangbo et al. [2019] trained an actuator network to produce reasonable predictions for the actuator of real robots since the actuator model of the simulator is not accurate enough.”
BackgroundSim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309 · cited 2×
“Tasks have different definitions in different scenarios, e.g., differences in reward functions [Finn et al., 2017a, Rothfuss et al., 2019], or parameters of dynamics [Peng et al., 2018, Zhang et al., 2018a].”
BackgroundMastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.01815 · Cited by 1,102 · cited 3×
“AlphaGo Zero [Silver et al., 2017b] is able to defeat professional human players without any human knowledge.”
BackgroundOffline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020) · arXiv · doi:10.48550/arxiv.2005.01643 · Cited by 824 · cited 1×
“Offline reinforcement learning studies the methodologies that enable the agent to directly learn an effective policy from an offline experience dataset without any interaction with the environment dynamics [Levine et al., 2020].”
BackgroundDyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777 · cited 1×
“Dynamic programming [Sutton and Barto, 2018], tabular Dyna [Sutton, 1990b, 1991], and prioritized sweeping [Moore and Atkeson, 1993] all belong to background planning methods.”
BackgroundMagnetic control of tokamak plasmas through deep reinforcement learning
Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de las Casas, D., et al. (2022) · Nature · doi:10.1038/s41586-021-04301-9 · Cited by 751 · cited 2×
“Even if the model can be unrealistic, it is possible to constrain the exploration (e.g., [Shi et al., 2019, Degrave et al., 2022]) to maintain the effectiveness of the learned policies.”
BackgroundSim-to-Real: Learning Agile Locomotion For Quadruped Robots
Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., & Vanhoucke, V. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.010 · Cited by 735
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., & Vanhoucke, V. (2018) · arXiv · doi:10.48550/arxiv.1804.10332 · Cited by 734
Learning Latent Dynamics for Planning from Pixels
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. (2018) · arXiv · doi:10.48550/arxiv.1811.04551 · Cited by 708
Solving Rubik's Cube with a Robot Hand
OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.07113 · Cited by 602
Dream to Control: Learning Behaviors by Latent Imagination
Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2019) · arXiv · doi:10.48550/arxiv.1912.01603 · Cited by 558
Model-Ensemble Trust-Region Policy Optimization
Kurutach, T., Clavera, I., Duan, Y., Tamar, A., & Abbeel, P. (2018) · arXiv · doi:10.48550/arxiv.1802.10592 · Cited by 537
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.12114 · Cited by 525 · cited 7×
“However, the deterministic transition model fails to capture the aleatoric uncertainty [Chua et al., 2018], which arises from the inherent stochasticities of the environment.”
BackgroundMethodInfluentialRL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., & Abbeel, P. (2016) · arXiv · doi:10.48550/arxiv.1611.02779 · Cited by 503
Self-citationTamar, A., Wu, Y., Thomas, G., Levine, S., & Abbeel, P. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/700 · Cited by 484
Sim-to-Real Robot Learning from Pixels with Progressive Nets
Rusu, A. A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1610.04286 · Cited by 385 · cited 1×
“…the agents have access to a simulator and deal with tasks from both simulation and reality, it relates to Sim2Real [Peng et al., 2018, Yu et al., 2017, Tan et al., 2018, Rusu et al., 2017, Chen et al., 2021a], which focuses on how to transfer a policy trained in the simulator to the real world.”
BackgroundLearning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning
Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., & Finn, C. (2018) · arXiv · doi:10.48550/arxiv.1803.11347 · Cited by 370
When to Trust Your Model: Model-Based Policy Optimization
Janner, M., Fu, J., Zhang, M., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1906.08253 · Cited by 351 · cited 9×
“In the recent analysis [Luo et al., 2018, Janner et al., 2019, Xu et al., 2020], the error of the learned transition model M θ is measured over the data distribution, and thus directly connects with the learning loss Eq.”
BackgroundMethodInfluentialMastering Atari with Discrete World Models
Hafner, D., Lillicrap, T., Norouzi, M., & Ba, J. (2020) · arXiv · doi:10.48550/arxiv.2010.02193 · Cited by 247 · cited 1×
“Hafner et al. [2021] further proposed DreamerV2 that supports the agent to learn purely from the model rollout data and achieve human-level performance on 55 Atari game tasks.”
BackgroundPreparing for the unknown: Learning a universal policy with online system identification
Yu, W., Tan, J., Karen Liu, C., & Turk, G. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.048 · Cited by 237
Preparing for the Unknown: Learning a Universal Policy with Online System Identification
Yu, W., Tan, J., Liu, C. K., & Turk, G. (2017) · arXiv · doi:10.48550/arxiv.1702.02453 · Cited by 237 · cited 1×
“…when the agents have access to a simulator and deal with tasks from both simulation and reality, it relates to Sim2Real [Peng et al., 2018, Yu et al., 2017, Tan et al., 2018, Rusu et al., 2017, Chen et al., 2021a], which focuses on how to transfer a policy trained in the simulator to the…”
BackgroundBenchmarking Model-Based Reinforcement Learning
Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., Zhang, S., Zhang, G., Abbeel, P., & Ba, J. (2019) · arXiv · doi:10.48550/arxiv.1907.02057 · Cited by 226
Improving Multi-Step Prediction of Learned Time Series Models
Venkatraman, A., Hebert, M., & Bagnell, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9590 · Cited by 198
A Differentiable Physics Engine for Deep Learning in Robotics
Degrave, J., Hermans, M., Dambre, J., & wyffels, F. (2019) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2019.00006 · Cited by 182 · cited 1×
“A dynamic model could be a neural network [Heess et al., 2015], Gaussian process [Deisenroth and Rasmussen, 2011], or a differentiable physics engine [Degrave et al., 2019].”
MethodSample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
Buckman, J., Hafner, D., Tucker, G., Brevdo, E., & Lee, H. (2018) · arXiv · doi:10.48550/arxiv.1807.01675 · Cited by 161 · cited 1×
“Stochastic ensemble value expansion (STEVE) [Buckman et al., 2018] further improves MVE by interpolating between different horizons H based on the uncertainty calculated using ensemble.”
BackgroundSearch on the Replay Buffer: Bridging Planning and Reinforcement Learning
Eysenbach, B., Salakhutdinov, R., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1906.05253 · Cited by 123
Goal-Conditioned Reinforcement Learning: Problems and Solutions
Liu, M., Zhu, M., & Zhang, W. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/770 · Cited by 115 · cited 2×
“Goal-conditioned reinforcement learning (GCRL), also named as goal-oriented RL [Liu et al., 2022], deals with the tasks where agents are expected to achieve different goals [Pitis et al.”
BackgroundSelf-citationFinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance
Liu, X.-Y. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3737257 · Cited by 90
Model-based Reinforcement Learning: A Survey
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2020) · arXiv · doi:10.48550/arxiv.2006.16712 · Cited by 86
Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.06175 · Cited by 63
On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
Zhang, B., Rajan, R., Pineda, L., Lambert, N., Biedenkapp, A., Chua, K., Hutter, F., & Calandra, R. (2021) · arXiv · doi:10.48550/arxiv.2102.13651 · Cited by 53 · cited 3×
“From the analysis of [Zhang et al., 2021b], the sample efficiency of MARL can be decomposed into two parts, i.e., dynamics sample complexity, which measures the amount of interactions with the real environment, and the opponent sample complexity, which measures the amount of interactions between the…”
BackgroundMethodA0C: Alpha Zero in Continuous Action Space
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2018) · arXiv · doi:10.48550/arxiv.1805.09613 · Cited by 34
Objective Mismatch in Model-based Reinforcement Learning
Lambert, N., Amos, B., Yadan, O., & Calandra, R. (2020) · arXiv · doi:10.48550/arxiv.2002.04523 · Cited by 33
Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
Seo, Y., Lee, K., Clavera, I., Kurutach, T., Shin, J., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2010.13303 · Cited by 20
Cited by
A multifaceted vision of the human-AI collaboration: a comprehensive review
Puerta-Beldarrain, M., Gómez-Carmona, O., Sánchez-Corcuera, R., Casado-Mansilla, D., López-de-Ipiña, D., & Chen, L. (2025) · IEEE Access · doi:10.1109/access.2025.3536095 · Cited by 73
Understanding World or Predicting Future? A Comprehensive Survey of World Models
Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., et al. (2026) · ACM Computing Surveys · doi:10.1145/3746449 · Cited by 169
“The learning labels are the trajectories derived from real interaction environments, also called the simulation data [164].”
Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?
Bengio, Y., Cohen, M., Fornasiere, D., Ghosn, J., Greiner, P., MacDermott, M., Mindermann, S., Oberman, A., Richardson, J., Richardson, O., et al. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i5.15569 · Cited by 41
Wu, Z., Yu, C., Chen, C., Hao, J., & Zhuo, H. H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26241 · Cited by 12
Metadata
- Authors
- Luo, F.-M., Xu, T., Lai, H., Chen, X.-H., Zhang, W., & Yu, Y.
- Venue
- Science China Information Sciences
- Published
- 2024-01-23
- Volume
- 67
- Issue
- 2
- ISSN
- 1869-1919 (ISSN-L)
- 1674-733X
- OpenAlex
- W4391259941
- Cited by
- 148
- References
- 45
- FWCI
- 19.79
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.