CoChat
Sign Up
PreprintCited by 351Reinforcement Learning in RoboticsAdvanced Bandit Algorithms ResearchAdaptive Dynamic Programming Control

When to Trust Your Model: Model-Based Policy Optimization

Open access
Janner, Michael, Fu, Justin, Zhang, Marvin, Levine, Sergey·2019·arXiv
DOI 10.48550/arxiv.1906.08253arXiv 1906.08253
Open this paper in CoChat — free accountOpen PDF

Abstract

Designing effective model-based reinforcement learning algorithms is difficult because the ease of data generation must be weighed against the bias of model-generated data. In this paper, we study the role of model usage in policy optimization both theoretically and empirically. We first formulate and analyze a model-based reinforcement learning algorithm with a guarantee of monotonic improvement at each step. In practice, this analysis is overly pessimistic and suggests that real off-policy data is always preferable to model-generated on-policy data, but we show that an empirical estimate of model generalization can be incorporated into such analysis to justify model usage. Motivated by this analysis, we then demonstrate that a simple procedure of using short model-generated rollouts branched from real data has the benefits of more complicated model-based algorithms without the usual pitfalls. In particular, this approach surpasses the sample efficiency of prior model-based methods, matches the asymptotic performance of the best model-free algorithms, and scales to horizons that cause other model-based methods to fail entirely.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1906.08253

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The ability to plan with uncertainty has allowed model-based algorithms to compete with their model-free alternatives.

    “This ability to plan with uncertainty has allowed model-based algorithms to compete with their model-free alternatives (Schrittwieser et al. 2020; Chua et al. 2018; Janner et al. 2019).”

    From Improving Controller Generalization with Dimensionless Markov Decision Processes (2025) · cites this paper as Janner et al. 2019

    The Introduction (p. 1) says MBPO retains the asymptotic performance of competitive model-free algorithms. Section 5 (p. 6) discusses handling model uncertainty, but Related Work (p. 2) says MBPO uses its model to learn a policy rather than for a planning routine. The paper therefore supports competitive performance, not the claim that MBPO achieved it through planning with uncertainty. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. Asadi, K., Misra, D., and Littman, M. Lipschitz continuity in model-based reinforcement learning. In International Conference on Machine Learning , 2018.
  2. Atkeson, C. G. and Schaal, S. Learning tasks from a single demonstration. In International Conference on Robotics and Automation , 1997.
  3. Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems . 2018.
  4. Clavera, I., Rothfuss, J., Schulman, J., Fujita, Y ., Asfour, T., and Abbeel, P. Model-based reinforcement learning via meta-policy optimization. In Conference on Robot Learning , 2018.
  5. Dean, S., Mania, H., Matni, N., Recht, B., and Tu, S. On the sample complexity of the linear quadratic regulator. arXiv preprint arXiv:1710.01688 , 2017. · arXiv
  6. Deisenroth, M. and Rasmussen, C. E. PILCO: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning , 2011.
  7. Depeweg, S., Hernández-Lobato, J. M., Doshi-Velez, F., and Udluft, S. Learning and policy search in stochastic dynamical systems with bayesian neural networks. In International Conference on Learning Representations , 2016.
  8. Draeger, A., Engell, S., and Ranke, H. Model predictive control using neural networks. IEEE Control Systems Magazine , 1995.
  9. Du, Y. and Narasimhan, K. Task-agnostic dynamics priors for deep reinforcement learning. In International Conference on Machine Learning , 2019.
  10. Ebert, F., Finn, C., Dasari, S., Xie, A., Lee, A. X., and Levine, S. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568 , 2018. · arXiv
  11. Farahmand, A.-M., Barreto, A., and Nikovski, D. Value-aware loss function for model-based reinforcement learning. In International Conference on Artificial Intelligence and Statistics , 2017.
  12. Feinberg, V., Wan, A., Stoica, I., Jordan, M. I., Gonzalez, J. E., and Levine, S. Model-based value estimation for efficient model-free reinforcement learning. In International Conference on Machine Learning , 2018.
  13. Gal, Y., McAllister, R., and Rasmussen, C. E. Improving PILCO with Bayesian neural network dynamics models. In ICML Workshop on Data-Efficient Machine Learning Workshop , 2016.
  14. Gu, S., Lillicrap, T., Sutskever, I., and Levine, S. Continuous deep Q-learning with model-based acceleration. In International Conference on Machine Learning , 2016.
  15. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning , 2018.
  16. Heess, N., Wayne, G., Silver, D., Lillicrap, T., Tassa, Y., and Erez, T. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems , 2015.
  17. Holland, G. Z., Talvitie, E. J., and Bowling, M. The effect of planning shape on dyna-style planning in high-dimensional state spaces. arXiv preprint arXiv:1806.01825 , 2018. · arXiv
  18. Kaiser, L., Babaeizadeh, M., Milos, P., Osinski, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowsi, P., Levine, S., Sepassi, R., Tucker, G., and Michalewski, H. Model-based reinforcement learning for Atari. arXiv preprint arXiv:1903.00374 , 2019. · arXiv
  19. Kalweit, G. and Boedecker, J. Uncertainty-driven imagination for continuous deep reinforcement learning. In Conference on Robot Learning , 2017.
  20. Kumar, V., Todorov, E., and Levine, S. Optimal control with learned local models: Application to dexterous manipulation. In International Conference on Robotics and Automation , 2016.
  21. Kurutach, T., Clavera, I., Duan, Y., Tamar, A., and Abbeel, P. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations , 2018.
  22. Levine, S. and Koltun, V. Guided policy search. In International Conference on Machine Learning , 2013.
  23. Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y ., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. In International Conference on Learning Representations , 2016.
  24. Luo, Y., Xu, H., Li, Y., Tian, Y., Darrell, T., and Ma, T. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In International Conference on Learning Representations , 2019.
  25. Nagabandi, A., Kahn, G., S. Fearing, R., and Levine, S. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In International Conference on Robotics and Automation , 2018.
  26. Oh, J., Guo, X., Lee, H., Lewis, R., and Singh, S. Action-conditional video prediction using deep networks in Atari games. In Advances in Neural Information Processing Systems , 2015.
  27. Oh, J., Singh, S., and Lee, H. Value prediction network. In Advances in Neural Information Processing Systems , 2017.
  28. Piché, A., Thomas, V., Ibrahim, C., Bengio, Y., and Pal, C. Probabilistic planning with sequential Monte Carlo methods. In International Conference on Learning Representations , 2019.
  29. Racanière, S., Weber, T., Reichert, D., Buesing, L., Guez, A., Jimenez Rezende, D., Puigdomènech Badia, A., Vinyals, O., Heess, N., Li, Y., Pascanu, R., Battaglia, P., Hassabis, D., Silver, D., and Wierstra, D. Imagination-augmented agents for deep reinforcement learning. In Advances in Neural Information Processing Systems . 2017.
  30. Rajeswaran, A., Ghotra, S., Levine, S., and Ravindran, B. EPOpt: Learning robust neural network policies using model ensembles. In International Conference on Learning Representations , 2017.
  31. Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International Conference on Machine Learning , 2015.
  32. Shalev-Shwartz, S. and Ben-David, S. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.
  33. Silver, D., van Hasselt, H., Hessel, M., Schaul, T., Guez, A., Harley, T., Dulac-Arnold, G., Reichert, D., Rabinowitz, N., Barreto, A., and Degris, T. The predictron: End-to-end learning and planning. In International Conference on Machine Learning , 2017.
  34. Sun, W., Gordon, G. J., Boots, B., and Bagnell, J. Dual policy iteration. In Advances in Neural Information Processing Systems , 2018.
  35. Sutton, R. S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In International Conference on Machine Learning , 1990.
  36. Szita, I. and Szepesvari, C. Model-based reinforcement learning with nearly tight exploration complexity bounds. In International Conference on Machine Learning , 2010.
  37. Talvitie, E. Model regularization for stable sample rollouts. In Conference on Uncertainty in Artificial Intelligence , 2014.
  38. Talvitie, E. Self-correcting models for model-based reinforcement learning. In AAAI Conference on Artificial Intelligence , 2016.
  39. Tamar, A., WU, Y., Thomas, G., Levine, S., and Abbeel, P. Value iteration networks. In Advances in Neural Information Processing Systems . 2016.
  40. Todorov, E., Erez, T., and Tassa, Y. MuJoCo: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems , 2012.

Cited by

Sorted by citations

Authors
Janner, M., Fu, J., Zhang, M., & Levine, S.
Affiliations
University of California, Berkeley (US)
ORCID
Levine, Sergey
Venue
arXiv
Publisher
arXiv
Published
2019
Volume
32
Pages
12498-12509
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1906.08253
arXiv
1906.08253
OpenAlex
W2950624398
Cited by
351
References
3
Citations by year
2026: 1 · 2025: 37 · 2024: 98 · 2023: 71 · 2022: 53 · 2021: 58 · 2020: 31 · 2019: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Funders
  • National Science Foundation (1651843, 1700697, 1700696)
  • Open Philanthropy Project
  • National Defense Science and Engineering Graduate
  • Office of Naval Research (W911NF-17-2-0181)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
Keywords
model-based reinforcement learning, policy optimization, model generalization, sample efficiency, long-horizon control

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review