The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning
Open accessAbstract
Model-free deep reinforcement learning algorithms have been shown to be capable of learning a wide range of robotic skills, but typically require a very large number of samples to achieve good performance. Model-based algorithms, in principle, can provide for much more efficient learning, but have proven difficult to extend to expressive, high-capacity models such as deep neural networks. In this work, we demonstrate that neural network dynamics models can in fact be combined with model predictive control (MPC) to achieve excellent sample complexity in a model-based reinforcement learning algorithm, producing stable and plausible gaits that accomplish various complex locomotion tasks. We further propose using deep neural network dynamics models to initialize a model-free learner, in order to combine the sample efficiency of model-based approaches with the high task-specific performance of model-free methods. We empirically demonstrate on MuJoCo locomotion tasks that our pure model-based approach trained on just random action data can follow arbitrary trajectories with excellent sample efficiency, and that our hybrid algorithm can accelerate model-free learning on high-speed benchmark tasks, achieving sample efficiency gains of 3-5× on swimmer, cheetah, hopper, and ant agents. Videos can be found at https://sites.google.com/view/mbmf.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160
Neural networks for control systems—A survey
Hunt, K. J., Sbarbaro, D., Żbikowski, R., & Gawthrop, P. J. (1992) · Automatica · doi:10.1016/0005-1098(92)90053-i · Cited by 1,864
(2000) · doi:10.2460/ajvr.67.2.323 · Cited by 1,621
End-to-End Training of Deep Visuomotor Policies
Levine, S., Finn, C., Darrell, T. J., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 1,344
Self-citationPILCO: A Model-Based and Data-Efficient Approach to Policy Search
Deisenroth, M. P., & Rasmussen, C. E. (2011) · Cambridge University Engineering Department Publications Database · Cited by 1,136
Benchmarking Deep Reinforcement Learning for Continuous Control
Duan, Y., Chen, X., Houthooft, R., Schulman, J., & Abbeel, P. (2016) · arXiv · doi:10.48550/arxiv.1604.06778 · Cited by 1,114
A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients
Grondman, I., Busoniu, L., Lopes, G. A. D., & Babuska, R. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2012.2218595 · Cited by 1,042
A Survey on Policy Search for Robotics
Deisenroth, M. P. (2011) · Foundations and Trends in Robotics · doi:10.1561/2300000021 · Cited by 798
Dyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777
Learning Stable Nonlinear Dynamical Systems With Gaussian Mixture Models
Khansari-Zadeh, S. M., & Billard, A. (2011) · IEEE Transactions on Robotics · doi:10.1109/tro.2011.2159412 · Cited by 771
ITERATIVE LINEAR QUADRATIC REGULATOR DESIGN FOR NONLINEAR BIOLOGICAL MOVEMENT SYSTEMS
(2004) · Proceedings of the First International Conference on Informatics in Control, Automation and Robotics · doi:10.5220/0001143902220229 · Cited by 543
Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics
Levine, S., & Abbeel, P. (2014) · Cited by 389
Self-citationLearning Continuous Control Policies by Stochastic Value Gradients
Heess, N., Wayne, G., Silver, D., Lillicrap, T., Tassa, Y., & Erez, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.09142 · Cited by 328
Model-Less Feedback Control of Continuum Manipulators in Constrained Environments
Yip, M. C., & Camarillo, D. B. (2014) · IEEE Transactions on Robotics · doi:10.1109/tro.2014.2309194 · Cited by 262
Control of Memory, Active Perception, and Action in MinecraftPreprint
Oh, J., Chockalingam, V., Singh, S. P., & Lee, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09128 · Cited by 243
Learning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning
Deisenroth, M., Rasmussen, C., & Fox, D. (2011) · Robotics: Science and Systems VII · doi:10.15607/rss.2011.vii.008 · Cited by 205
From Pixels to Torques: Policy Learning with Deep Dynamical ModelsPreprint
Wahlström, N., Schön, T. B., & Deisenroth, M. P. (2015) · arXiv (Cornell University) · doi:10.25561/21120 · Cited by 134
(1993) · Springer eBooks · doi:10.1007/978-1-4615-3180-7 · Cited by 125
Sample-based learning and search with permanent and transient memories
Silver, D., Sutton, R. S., & Müller, M. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390278 · Cited by 109
Robust constrained model predictive control
Richards, A. G. (2005) · Cited by 96
Minimax Differential Dynamic Programming: An Application to Robust Biped Walking
Morimoto, J., & Atkeson, C. G. (2018) · doi:10.1184/r1/6555545 · Cited by 77
Prediction and Control with Temporal Segment ModelsPreprint
Mishra, N., Abbeel, P., & Mordatch, I. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.04070 · Cited by 45
Learning legged swimming gaits from experience
Meger, D., Higuera, J. C. G., Xu, A., et al. (2015) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2015.7139509 · Cited by 35
Toward fast policy search for learning legged locomotion
Deisenroth, M. P., Calandra, R., Andre Seyfarth, A., & Peters, J. (2012) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2012.6385955 · Cited by 32
Strengths, Weaknesses, and Combinations of Model-based and Model-free Reinforcement Learning
Asadi, K. (2016) · ERA: Education and Research Archive (University of Alberta) · doi:10.7939/r3c24qw38 · Cited by 8
Research works printed but not linked: 13
- rao, 2009, A survey of numerical methods for optimal control, Advances in the Astronautical Sciences
- gal, 2016, Improving PILCO with bayesian neural network dynamics models, Data-Efficient Machine Learning workshop
- ross, 2011, A reduction of imitation learning and structured prediction to no-regret online learning, AISTATS
- schulman, 2016, High-dimensional continuous control using generalized advantage estimation, ICLRE
- gu, 2017, Q-Prop: sample-efficient policy gradient with an off-policy critic, ICLRE
- schulman, 2015, Trust region policy optimization, ICML
- watter, 2015, Embed to control: a locally linear latent dynamics model for control from raw images, NIPS
- lioutikov, 2014, Sample-based information-theoretic stochastic optimal control, ICRA
- brockman, 2016, OpenAI Gym
- gu, 2016, Continuous deep Q-learning with model-based acceleration, ICML
- ko, 2008, GP-BayesFilters: Bayesisn filtering using gaussian process prediction and observation models, IROS
- boedecker, 2014, Approximate real-time optimal control based on sparse gaussian process models, Adprl
- kingma, 2014, Adam: A method for stochastic optimization, ICLRE
Cited by
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733
A Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
Nguyen, T. T., Nguyen, N. D., & Nahavandi, S. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2977374 · Cited by 1,306
Transfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020) · arXiv · doi:10.48550/arxiv.2005.01643 · Cited by 824
Self-citation
Metadata
- Authors
- Nagabandi, A., Kahn, G., Fearing, R. S., & Levine, S.
- Venue
- 2018 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2018-05
- Pages
- 7559-7566
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2743381431
- Cited by
- 848
- References
- 30
- FWCI
- 53.26
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.