Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 848

The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.

Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

Open access
Nagabandi, Anusha, Kahn, Gregory, Fearing, Ronald S., Levine, Sergey·2018·2018 IEEE International Conference on Robotics and Automation (ICRA)

Abstract

Model-free deep reinforcement learning algorithms have been shown to be capable of learning a wide range of robotic skills, but typically require a very large number of samples to achieve good performance. Model-based algorithms, in principle, can provide for much more efficient learning, but have proven difficult to extend to expressive, high-capacity models such as deep neural networks. In this work, we demonstrate that neural network dynamics models can in fact be combined with model predictive control (MPC) to achieve excellent sample complexity in a model-based reinforcement learning algorithm, producing stable and plausible gaits that accomplish various complex locomotion tasks. We further propose using deep neural network dynamics models to initialize a model-free learner, in order to combine the sample efficiency of model-based approaches with the high task-specific performance of model-free methods. We empirically demonstrate on MuJoCo locomotion tasks that our pure model-based approach trained on just random action data can follow arbitrary trajectories with excellent sample efficiency, and that our hybrid algorithm can accelerate model-free learning on high-speed benchmark tasks, achieving sample efficiency gains of 3-5× on swimmer, cheetah, hopper, and ant agents. Videos can be found at https://sites.google.com/view/mbmf.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/icra.2018.8463189

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 30 references

Research works printed but not linked: 13

  1. rao, 2009, A survey of numerical methods for optimal control, Advances in the Astronautical Sciences
  2. gal, 2016, Improving PILCO with bayesian neural network dynamics models, Data-Efficient Machine Learning workshop
  3. ross, 2011, A reduction of imitation learning and structured prediction to no-regret online learning, AISTATS
  4. schulman, 2016, High-dimensional continuous control using generalized advantage estimation, ICLRE
  5. gu, 2017, Q-Prop: sample-efficient policy gradient with an off-policy critic, ICLRE
  6. schulman, 2015, Trust region policy optimization, ICML
  7. watter, 2015, Embed to control: a locally linear latent dynamics model for control from raw images, NIPS
  8. lioutikov, 2014, Sample-based information-theoretic stochastic optimal control, ICRA
  9. brockman, 2016, OpenAI Gym
  10. gu, 2016, Continuous deep Q-learning with model-based acceleration, ICML
  11. ko, 2008, GP-BayesFilters: Bayesisn filtering using gaussian process prediction and observation models, IROS
  12. boedecker, 2014, Approximate real-time optimal control based on sparse gaussian process models, Adprl
  13. kingma, 2014, Adam: A method for stochastic optimization, ICLRE

Cited by

Metadata

Authors
Nagabandi, A., Kahn, G., Fearing, R. S., & Levine, S.
Venue
2018 IEEE International Conference on Robotics and Automation (ICRA)
Publisher
IEEE
Published
2018-05
Pages
7559-7566
ISSN
1050-4729 (ISSN-L)
OpenAlex
W2743381431
Cited by
848
References
30
FWCI
53.26
Type
conference paper
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.