The DOI matches its Crossref record; 43 of the 45 references it cites were found in a registry or an index.
Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search
Open accessAbstract
Model predictive control (MPC) is an effective method for controlling robotic systems, particularly autonomous aerial vehicles such as quadcopters. However, application of MPC can be computationally demanding, and typically requires estimating the state of the system, which can be challenging in complex, unstructured environments. Reinforcement learning can in principle forego the need for explicit state estimation and acquire a policy that directly maps sensor readings to actions, but is difficult to apply to unstable systems that are liable to fail catastrophically during training before an effective policy has been found. We propose to combine MPC with reinforcement learning in the framework of guided policy search, where MPC is used to generate data at training time, under full state observations provided by an instrumented training environment. This data is used to train a deep neural network policy, which is allowed to access only the raw observations from the vehicle's onboard sensors. After training, the neural network policy can successfully control the robot without knowledge of the full state, and at a fraction of the computational cost of MPC. We evaluate our method by learning obstacle avoidance policies for a simulated quadrotor, using simulated onboard sensors and no explicit state estimation at test time.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The frequent sensing and re-planning allow the robot to react dynamically to changes in the environment, while still making more intelligent choices than a purely reactive system (Erez et al., 2011; Lenz et al., 2015; Zhang et al., 2016; Liu et al., 2017; Williams et al., 2017; Gillespie et al., 2018; Amos et al., 2019; Kabzan et al., 2019).”
From Lessons for Robotics From the Control Architecture of the Octopus (2022) · cites this paper as Zhang et al., 2016
In the citing paper
“AField-Programmable Gate Array (FPGA) is another viable solution, showing great improvements in both performance and power consumption in deep learning applications.”
From A survey of deep learning techniques for autonomous driving (2020) · cites this paper as [170]
In the citing paper
“Currently, the largest portion of papers in deep learning for selfdriving cars are focused mainly on perception and End2End learning [81, 124].”
From A survey of deep learning techniques for autonomous driving (2020) · cites this paper as [124]
In the citing paper
“Finally, realtime requirements have to be fulfilled for processing the large amounts of data gathered from the car's sensors suite, as well as for updating the parameters of deep learning systems over high-speed communication lines [170].”
From A survey of deep learning techniques for autonomous driving (2020) · cites this paper as [170]
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“We used default values for all learning parameters as presented in [14].”
Method(2010) · International Conference on Machine Learning · Cited by 13,995
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · cited 1×
“More recently, deep neural network policies have been used for tasks ranging from robotic control [22], [23] to video game playing [27].”
MethodMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · cited 2×
“However, our derivation largely follows prior work [41].”
MethodResultReinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160 · cited 1×
“An overview of recent reinforcement learning methods in robotics can be found in a recent survey paper [15].”
BackgroundEnd-to-End Training of Deep Visuomotor PoliciesPreprint
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 2,641 · cited 10×
“In this paper, we use the BADMM version, which specifies the following objective for trajectory optimization: min pi(τ) T∑ t=1 Epi(xt,ut) [ `(xt,ut)− uTt λiµt + νitDKL ( pi(ut|xt)‖πθ(ut|xt) )] (1) where λµt is a Lagrange multiplier on the mean action, and the third term is a KL-divergence penalty.”
BackgroundMethodInfluentialSelf-citationA Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941
Neural networks for control systems—A survey
Hunt, K. J., Sbarbaro, D. G., Żbikowski, R., & Gawthrop, P. J. (1992) · Automatica · doi:10.1016/0005-1098(92)90053-i · Cited by 1,874 · cited 1×
“neural networks have been used for control for decades [12], [33], limitations on computation and algorithms have made large neural network policies very difficult to learn.”
MethodRobust model predictive control of constrained linear systems with bounded disturbances
Mayne, D. Q., Seron, M. M., & Raković, S. V. (2004) · Automatica · doi:10.1016/j.automatica.2004.08.019 · Cited by 1,680 · cited 1×
“Model predictive control (MPC) is an effective and reliable method for controlling robotic systems, particularly autonomous aerial vehicles such as quadcopters, because of its robustness to moderate model errors [26], ability to use high-level objectives [41], and relative simplicity.”
MethodALVINN, an autonomous land vehicle in a neural network
Pomerleau, D. (2018) · doi:10.1184/r1/6603146 · Cited by 1,369
ALVINN: An autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6603146.v1 · Cited by 1,340
A Survey on Policy Search for Robotics
Deisenroth, M. P. (2011) · now publishers, Inc. eBooks · doi:10.1561/2300000021 · Cited by 845 · cited 2×
“On the other end of the spectrum from such model-based methods, reinforcement learning (RL) aims to directly learn control policies that map observations to controls [5].”
BackgroundReinforcement learning of motor skills with policy gradients
Peters, J., & Schaal, S. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.02.003 · Cited by 831 · cited 1×
“While this method is not as general as policy gradient RL algorithms, which can optimize arbitrary parameterized policies with arbitrary observations under unknown dynamics [32], it is fast, simple, and often effective [39].”
MethodSynthesis and stabilization of complex behaviors through online trajectory optimization
Tassa, Y., Erez, T., & Todorov, E. V. (2012) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2012.6386025 · Cited by 815 · cited 2×
“While this method is not as general as policy gradient RL algorithms, which can optimize arbitrary parameterized policies with arbitrary observations under unknown dynamics [32], it is fast, simple, and often effective [39].”
MethodAn integral predictive/nonlinear H ∞ control structure for a quadrotor helicopter
Raffo, G. V., Ortega, M. G., & Rubio, F. R. (2009) · Automatica · doi:10.1016/j.automatica.2009.10.018 · Cited by 795 · cited 1×
“quently used to control autonomous aerial vehicles such as quadrotors [38], [35], [34], [3], [2], [4], [29].”
Learning and generalization of motor skills by learning from demonstration
Pastor, P., Hoffmann, H., Asfour, T., & Schaal, S. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152385 · Cited by 702 · cited 1×
“RL has been used to train robotic controllers for games such as ball-in-cup and table tennis [16], manipulation [31], [6], and robotic locomotion [17], [40], [10], [8].”
MethodITERATIVE LINEAR QUADRATIC REGULATOR DESIGN FOR NONLINEAR BIOLOGICAL MOVEMENT SYSTEMS
Li, W., & Todorov, E. V. (2004) · International Conference on Informatics in Control, Automation and Robotics (ICINCO) · doi:10.5220/0001143902220229 · Cited by 589
Policy gradient reinforcement learning for fast quadrupedal locomotion
Kohl, N., & Stone, P. (2004) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2004.1307456 · Cited by 586 · cited 1×
“RL has been used to train robotic controllers for games such as ball-in-cup and table tennis [16], manipulation [31], [6], and robotic locomotion [17], [40], [10], [8].”
MethodAn Application of Reinforcement Learning to Aerobatic Helicopter Flight
Abbeel, P., Coates, A., Quigley, M., & Ng, A. Y. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7503.003.0006 · Cited by 551 · cited 1×
“While such methods have been successfully applied to aerial vehicles [1], [28], the requirement to be able to acquire an accurate model means that these methods share many of the challenges of MPC methods.”
MethodSelf-citationDifferential dynamic programming
DENHAM, W. F. (1971) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1971.1099760 · Cited by 530
Differential dynamic programming
Jacobson, D. H., & Mayne, D. Q. (1970) · CERN Document Server (European Organization for Nuclear Research) · Cited by 510
Levine, S., & Koltun, V. (2013) · Cited by 481
Self-citationLearning monocular reactive UAV control in cluttered natural environments
Ross, S., Melik-Barkhudarov, N., Shankar, K. S., Wendel, A., Dey, D., Bagnell, J. A., & Hebert, M. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6630809 · Cited by 410 · cited 1×
“While alternative learning methods, such as learning from demonstration [36], [37], can address this”
Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics
Levine, S., & Abbeel, P. (2014) · Cited by 389
Self-citationModel predictive quadrotor control: attitude, altitude and position experimental studies
Alexis, K., Nikolakopoulos, G., & Tzes, A. P. (2012) · IET Control Theory and Applications · doi:10.1049/iet-cta.2011.0348 · Cited by 372 · cited 1×
“quently used to control autonomous aerial vehicles such as quadrotors [38], [35], [34], [3], [2], [4], [29].”
DeepMPC: Learning Deep Latent Features for Model Predictive Control
Lenz, I., Knepper, R. A., & Saxena, A. (2015) · Robotics: Science and Systems · doi:10.15607/rss.2015.xi.012 · Cited by 349 · cited 1×
“Deep neural networks have also been used to learn models for MPC [18].”
MethodAugugliaro, F., Schoellig, A. P., & D’Andrea, R. (2012) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2012.6385823 · Cited by 341 · cited 1×
“quently used to control autonomous aerial vehicles such as quadrotors [38], [35], [34], [3], [2], [4], [29].”
Vision-based autonomous mapping and exploration using a quadrotor MAV
Fraundorfer, F., Heng, L., Honegger, D., Lee, G. H., Meier, L., Tanskanen, P., & Pollefeys, M. (2012) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2012.6385934 · Cited by 319 · cited 1×
“This challenge is conventionally addressed by using localization and mapping algorithms to map out the environment [9] and then optimizing trajectories under the resulting map [11].”
Learning contact-rich manipulation skills with guided policy search
Levine, S., Wagener, N., & Abbeel, P. (2015) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2015.7138994 · Cited by 282 · cited 2×
“Guided policy search has been applied to locomotion [21], robotic manipulation [22], and visionbased robotic control [23], but all prior applications rely on an offline trajectory optimization phase to generate the controller that is then executed on the real system.”
MethodSelf-citationStochastic policy gradient reinforcement learning on a simple 3D biped
Tedrake, R., Zhang, T., & Seung, H. S. (2005) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2004.1389841 · Cited by 262 · cited 1×
“RL has been used to train robotic controllers for games such as ball-in-cup and table tennis [16], manipulation [31], [6], and robotic locomotion [17], [40], [10], [8].”
MethodSelf-citationThe true role of accelerometer feedback in quadrotor control
Martin, P., & Salaun, E. (2010) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2010.5509980 · Cited by 243 · cited 1×
“The dynamics followed the formulation described by Martin and Salaun [25].”
MethodEfficient mixed-integer planning for UAVs in cluttered environments
Deits, R., & Tedrake, R. (2015) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2015.7138978 · Cited by 228 · cited 1×
“principle, a wide variety of methods can be used to generate guiding samples, and more sophisticated methods might afford superior robustness and obstacle avoidance [7].”
BackgroundLearning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning
Deisenroth, M. P., Rasmussen, C. E., & Fox, D. (2011) · Robotics: Science and Systems · doi:10.15607/rss.2011.vii.008 · Cited by 208 · cited 1×
“RL has been used to train robotic controllers for games such as ball-in-cup and table tennis [16], manipulation [31], [6], and robotic locomotion [17], [40], [10], [8].”
MethodLearning CPG-based Biped Locomotion with a Policy Gradient Method: Application to a Humanoid Robot
Endo, G., Morimoto, J., Matsubara, T., Nakanishi, J., & Cheng, G. (2008) · The International Journal of Robotics Research · doi:10.1177/0278364907084980 · Cited by 206 · cited 1×
“RL has been used to train robotic controllers for games such as ball-in-cup and table tennis [16], manipulation [31], [6], and robotic locomotion [17], [40], [10], [8].”
MethodNonlinear model predictive tracking control for rotorcraft-based unmanned aerial vehicles
Kim, H. J., Shim, D. H., & Sastry, S. (2002) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.2002.1024483 · Cited by 196 · cited 1×
“quently used to control autonomous aerial vehicles such as quadrotors [38], [35], [34], [3], [2], [4], [29].”
Bouffard, P., Aswani, A. J., & Tomlin, C. J. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6225035 · Cited by 183 · cited 1×
“quently used to control autonomous aerial vehicles such as quadrotors [38], [35], [34], [3], [2], [4], [29].”
Reinforcement Learning to adjust Robot Movements to New Situations
Kober, J., Öztop, E., & Peters, J. (2010) · Robotics: Science and Systems · doi:10.15607/rss.2010.vi.005 · Cited by 160
Learning Complex Neural Network Policies with Trajectory Optimization
Levine, S., & Koltun, V. (2014) · Cited by 115
Self-citationA model predictive controller for quadrocopter state interception
Mueller, M. W., & D’Andrea, R. (2013) · European Control Conference (ECC) · doi:10.23919/ecc.2013.6669415 · Cited by 112 · cited 1×
“quently used to control autonomous aerial vehicles such as quadrotors [38], [35], [34], [3], [2], [4], [29].”
Autonomous obstacle avoidance and maneuvering on a vision-guided MAV using on-board processing
Heng, L., Meier, L., Tanskanen, P., Fraundorfer, F., & Pollefeys, M. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5980095 · Cited by 103 · cited 1×
“This challenge is conventionally addressed by using localization and mapping algorithms to map out the environment [9] and then optimizing trajectories under the resulting map [11].”
Robust constrained model predictive control
Richards, A. G. (2005) · Cited by 96
Iterative learning of feed-forward corrections for high-performance tracking
Mueller, F. L., Schoellig, A. P., & D’Andrea, R. (2012) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2012.6385647 · Cited by 49 · cited 1×
“While such methods have been successfully applied to aerial vehicles [1], [28], the requirement to be able to acquire an accurate model means that these methods share many of the challenges of MPC methods.”
MethodFast biped walking with a reflexive controller and real-time policy searching
Geng, T., Porr, B., & Wörgötter, F. (2005) · neural information processing systems · Cited by 18
Flagged first, then by citations
2 of the 45 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Efficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215
“• Robotics DNNs have been successful in the domain of robotic tasks such as grasping with a robotic arm [32], motion planning for ground robots [33], visual navigation [4, 34], control to stabilize a quadcopter [35] and driving strategies for autonomous vehicles [36].”
A survey of deep learning techniques for autonomous driving
Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) · Journal of Field Robotics · doi:10.1002/rob.21918 · Cited by 1,809
“rently, the largest portion of papers in deep learning for selfdriving cars are focused mainly on perception and End2End learning [81, 124].”
BackgroundInfluentialData-Driven Science and Engineering
Brunton, S. L., & Kutz, J. N. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108380690 · Cited by 1,269
Socially aware motion planning with deep reinforcement learning
Chen, Y. F., Everett, M., Liu, M., & How, J. P. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202312 · Cited by 743
“While several recent works have applied deep RL to motion planning [22], [23], they are mainly focused on single agent navigation in unknown static environments, and with an emphasis on computing control inputs directly from raw”
BackgroundDecentralized non-communicating multiagent collision avoidance with deep reinforcement learning
Chen, Y. F., Liu, M., Everett, M., & How, J. P. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989037 · Cited by 689
“While applications of RL are typically limited to discrete, low-dimensional domains, recent advances in Deep RL [20]– [23] have demonstrated human-level performance in complex, high-dimensional spaces.”
Background
Metadata
- Authors
- Zhang, T., Kahn, G., Levine, S., & Abbeel, P.
- Affiliations
- University of California, Berkeley (US)
- Venue
- 2016 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2016-05
- Pages
- 528-535
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W1923344279
- Cited by
- 452
- References
- 45
- Citations by year
- 2026: 10 · 2025: 18 · 2024: 37 · 2023: 33 · 2022: 54 · 2021: 53 · 2020: 65 · 2019: 93 · 2018: 42 · 2017: 39 · 2016: 8
- FWCI
- 32.08
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (N66001)
- Army Research Office
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Physical Sciences › Engineering › Control and Systems Engineering › Advanced Control Systems Optimization
- Keywords
- model predictive control, deep neural network policies, autonomous aerial vehicles, quadrotor control, obstacle avoidance, reinforcement learning, state estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
