CoChat
Sign Up
PreprintCited by 50Reinforcement Learning in RoboticsAdvanced Control Systems OptimizationAdaptive Dynamic Programming Control

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Temporal Difference Learning for Model Predictive Control

Open access
Hansen, Nicklas, Wang, Xiaolong, Su, Hao·2022·arXiv
DOI 10.48550/arxiv.2203.04955arXiv 2203.04955
Open this paper in CoChat — free accountOpen PDF

Abstract

Data-driven model predictive control has two key advantages over model-free methods: a potential for improved sample efficiency through model learning, and better performance as computational budget for planning increases. However, it is both costly to plan over long horizons and challenging to obtain an accurate model of the environment. In this work, we combine the strengths of model-free and model-based methods. We use a learned task-oriented latent dynamics model for local trajectory optimization over a short horizon, and use a learned terminal value function to estimate long-term return, both of which are learned jointly by temporal difference learning. Our method, TD-MPC, achieves superior sample efficiency and asymptotic performance over prior work on both state and image-based continuous control tasks from DMControl and Meta-World. Code and video results are available at https://nicklashansen.github.io/td-mpc.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2203.04955

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent works combining learning methods with MPC rely on explicit, hand-crafted dynamics models and cost functions.

    “Though recent works combining learning methods with MPC have demonstrated real-time control capabilities in narrow domains [53, 21], they rely on explicit, hand-crafted dynamics models and cost functions.”

    From Real-Time Execution of Action Chunking Flow Policies (2025) · cites this paper as [21]

    Section 3 (p. 3) describes TD-MPC as combining MPC with a learned latent dynamics model and learned reward model. The Introduction (p. 1) also says its terminal value function is learned. As an example of the cited recent work, it contradicts the claim that such methods rely on explicit, hand-crafted dynamics models and cost functions. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems , 2021.
  2. Argenson, A. and Dulac-Arnold, G. Model-based offline planning. ArXiv , abs/2008.05556, 2021. · arXiv
  3. Bhardwaj, M., Handa, A., Fox, D., and Boots, B. Information theoretic model predictive q-learning. ArXiv , abs/2001.02153, 2020. · arXiv
  4. Bhardwaj, M., Choudhury, S., and Boots, B. Blending mpc & value function approximation for efficient reinforcement learning. ArXiv , abs/2012.05909, 2021. · arXiv
  5. Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H. Sample-efficient reinforcement learning with stochastic ensemble value expansion. In NeurIPS , 2018.
  6. Chen, X. and He, K. Exploring simple siamese representation learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 1574515753, 2021.
  7. Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In NeurIPS , 2018.
  8. Clavera, I., Fu, Y., and Abbeel, P. Model-augmented actor-critic: Backpropagating through paths. ArXiv , abs/2005.08068, 2020. · arXiv
  9. Ebert, F., Finn, C., Dasari, S., Xie, A., Lee, A. X., and Levine, S. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. ArXiv , abs/1812.00568, 2018. · arXiv
  10. Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S., and Kavukcuoglu, K. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. ArXiv , abs/1802.01561, 2018. · arXiv
  11. Fujimoto, S., Hoof, H. V., and Meger, D. Addressing function approximation error in actor-critic methods. ArXiv , abs/1802.09477, 2018. · arXiv
  12. Ha, D. and Schmidhuber, J. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems 31 , pp. 2451-2463. Curran Associates, Inc., 2018.
  13. Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. Soft actor-critic algorithms and applications. ArXiv , abs/1812.05905, 2018. · arXiv
  14. Hafez, M. B., Weber, C., Kerzel, M., and Wermter, S. Curious meta-controller: Adaptive alternation between modelbased and model-free control in deep reinforcement learning. 2019 International Joint Conference on Neural Networks (IJCNN) , pp. 1-8, 2019.
  15. Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning , pp. 2555-2565, 2019.
  16. Hafner, D., Lillicrap, T., Norouzi, M., and Ba, J. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193 , 2020a. · arXiv
  17. Hafner, D., Lillicrap, T. P., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. ArXiv , abs/1912.01603, 2020b. · arXiv
  18. Hansen, N. and Wang, X. Generalization in reinforcement learning by soft data augmentation. In International Conference on Robotics and Automation (ICRA) , 2021.
  19. Hansen, N., Jangir, R., Sun, Y., Aleny` a, G., Abbeel, P., Efros, A. A., Pinto, L., and Wang, X. Self-supervised policy adaptation during deployment. In International Conference on Learning Representations (ICLR) , 2021.
  20. Hatch, N. and Boots, B. The value of planning for infinitehorizon model predictive control. 2021 IEEE International Conference on Robotics and Automation (ICRA) , pp. 7372-7378, 2021.
  21. Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-based policy optimization. ArXiv , abs/1906.08253, 2019. · arXiv
  22. Kaiser, L., Babaeizadeh, M., Milos, P., Osinski, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., Sepassi, R., Tucker, G., and Michalewski, H. Model-based reinforcement learning for atari. ArXiv , abs/1903.00374, 2020. · arXiv
  23. Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., and Levine, S. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. ArXiv , abs/1806.10293, 2018. · arXiv
  24. Kalashnikov, D., Varley, J., Chebotar, Y., Swanson, B., Jonschkowski, R., Finn, C., Levine, S., and Hausman, K. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. ArXiv , abs/2104.08212, 2021. · arXiv
  25. Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel : Model-based offline reinforcement learning. ArXiv , abs/2005.05951, 2020. · arXiv
  26. Kostrikov, I., Yarats, D., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. International Conference on Learning Representations , 2020.
  27. Lillicrap, T., Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. CoRR , abs/1509.02971, 2016. · arXiv
  28. Lowrey, K., Rajeswaran, A., Kakade, S. M., Todorov, E., and Mordatch, I. Plan online, learn offline: Efficient learning and exploration via model-based control. ArXiv , abs/1811.01848, 2019. · arXiv
  29. Margolis, G., Chen, T., Paigwar, K., Fu, X., Kim, D., Kim, S., and Agrawal, P. Learning to jump from pixels. In CoRL , 2021.
  30. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013. · arXiv
  31. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In ICML , 2016.
  32. Morgan, A. S., Nandha, D., Chalvatzaki, G., D'Eramo, C., Dollar, A. M., and Peters, J. Model predictive actor-critic: Accelerating robot skill acquisition with deep reinforcement learning. 2021 IEEE International Conference on Robotics and Automation (ICRA) , pp. 6672-6678, 2021.
  33. Nagabandi, A., Kahn, G., Fearing, R. S., and Levine, S. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. 2018 IEEE International Conference on Robotics and Automation (ICRA) , pp. 7559-7566, 2018.
  34. Negenborn, R. R., De Schutter, B., Wiering, M. A., and Hellendoorn, H. Learning-based model predictive control for markov decision processes. IFAC Proceedings Volumes , 38(1):354-359, 2005. 16th IFAC World Congress.
  35. Nguyen, T. D., Shu, R., Pham, T., Bui, H. H., and Ermon, S. Temporal predictive coding for model-based planning in latent space. In ICML , 2021.
  36. Pong, V. H., Gu, S. S., Dalal, M., and Levine, S. Temporal difference models: Model-free deep rl for model-based control. ArXiv , abs/1802.09081, 2018. · arXiv
  37. Pourchot, A. and Sigaud, O. Cem-rl: Combining evolutionary and gradient-based methods for policy search. ArXiv , abs/1810.01222, 2019. · arXiv
  38. Rubinstein, R. Y. Optimization of computer simulation models with rare events. European Journal of Operational Research , 99:89-112, 1997.
  39. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. CoRR , abs/1511.05952, 2016. · arXiv
  40. Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. Planning to explore via self-supervised world models. ArXiv , abs/2005.05960, 2020. · arXiv
  41. Shao, L., You, Y., Yan, M., Sun, Q., and Bohg, J. Grac: Selfguided and self-regularized actor-critic. arXiv preprint arXiv:2009.08973 , 2020. · arXiv
  42. Sikchi, H., Zhou, W., and Held, D. Learning off-policy with online planning. In Conference on Robot Learning , pp. 1622-1633. PMLR, 2022.
  43. Srinivas, A., Laskin, M., and Abbeel, P. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.04136 , 2020. · arXiv
  44. Tassa, Y., Erez, T., and Todorov, E. Synthesis and stabilization of complex behaviors through online trajectory optimization. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pp. 4906-4913, 2012.
  45. Wang, T. and Ba, J. Exploring model-based planning with policy networks. ArXiv , abs/1906.08649, 2020. · arXiv
  46. Williams, G., Aldrich, A., and Theodorou, E. A. Model predictive path integral control using covariance variable importance sampling. ArXiv , abs/1509.01149, 2015. · arXiv
  47. Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645 , 2021. · arXiv
  48. Ye, W., Liu, S., Kurutach, T., Abbeel, P., and Gao, Y . Mastering atari games with limited data. ArXiv , abs/2111.00210, 2021. · arXiv
  49. Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL) , 2019.
  50. Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. ArXiv , abs/2005.13239, 2020. · arXiv
  51. Zhang, M., Vikram, S., Smith, L., Abbeel, P., Johnson, M. J., and Levine, S. Solar: Deep structured latent representations for model-based reinforcement learning. ArXiv , abs/1808.09105, 2018. · arXiv
  52. Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence , volume 3, 2008.

Cited by

Sorted by citations

Metadata

Authors
Hansen, N., Wang, X., & Su, H.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2203.04955
arXiv
2203.04955
OpenAlex
W4221164079
Cited by
50
References
1
Citations by year
2025: 10 · 2024: 27 · 2023: 11 · 2022: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Advanced Control Systems Optimization
  • Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
Keywords
model predictive control, temporal difference learning, data-driven control, latent dynamics models, sample efficiency, continuous control, visual servoing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review