The DOI matches its Crossref record; 43 of the 45 references it cites were found in a registry or an index.
Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search
Open accessAbstract
Model predictive control (MPC) is an effective method for controlling robotic systems, particularly autonomous aerial vehicles such as quadcopters. However, application of MPC can be computationally demanding, and typically requires estimating the state of the system, which can be challenging in complex, unstructured environments. Reinforcement learning can in principle forego the need for explicit state estimation and acquire a policy that directly maps sensor readings to actions, but is difficult to apply to unstable systems that are liable to fail catastrophically during training before an effective policy has been found. We propose to combine MPC with reinforcement learning in the framework of guided policy search, where MPC is used to generate data at training time, under full state observations provided by an instrumented training environment. This data is used to train a deep neural network policy, which is allowed to access only the raw observations from the vehicle's onboard sensors. After training, the neural network policy can successfully control the robot without knowledge of the full state, and at a fraction of the computational cost of MPC. We evaluate our method by learning obstacle avoidance policies for a simulated quadrotor, using simulated onboard sensors and no explicit state estimation at test time.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The frequent sensing and re-planning allow the robot to react dynamically to changes in the environment, while still making more intelligent choices than a purely reactive system (Erez et al., 2011; Lenz et al., 2015; Zhang et al., 2016; Liu et al., 2017; Williams et al., 2017; Gillespie et al., 2018; Amos et al., 2019; Kabzan et al., 2019).”
From Lessons for Robotics From the Control Architecture of the Octopus (2022) · cites this paper as Zhang et al., 2016
In the citing paper
“AField-Programmable Gate Array (FPGA) is another viable solution, showing great improvements in both performance and power consumption in deep learning applications.”
From A survey of deep learning techniques for autonomous driving (2020) · cites this paper as [170]
In the citing paper
“Currently, the largest portion of papers in deep learning for selfdriving cars are focused mainly on perception and End2End learning [81, 124].”
From A survey of deep learning techniques for autonomous driving (2020) · cites this paper as [124]
In the citing paper
“Finally, realtime requirements have to be fulfilled for processing the large amounts of data gathered from the car's sensors suite, as well as for updating the parameters of deep learning systems over high-speed communication lines [170].”
From A survey of deep learning techniques for autonomous driving (2020) · cites this paper as [170]
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“We used default values for all learning parameters as presented in [14].”
Method(2010) · International Conference on Machine Learning · Cited by 13,995
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · cited 1×
“More recently, deep neural network policies have been used for tasks ranging from robotic control [22], [23] to video game playing [27].”
MethodMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · cited 2×
“However, our derivation largely follows prior work [41].”
MethodResultReinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160 · cited 1×
“An overview of recent reinforcement learning methods in robotics can be found in a recent survey paper [15].”
BackgroundEnd-to-End Training of Deep Visuomotor PoliciesPreprint
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 2,641 · cited 10×
“In this paper, we use the BADMM version, which specifies the following objective for trajectory optimization: min pi(τ) T∑ t=1 Epi(xt,ut) [ `(xt,ut)− uTt λiµt + νitDKL ( pi(ut|xt)‖πθ(ut|xt) )] (1) where λµt is a Lagrange multiplier on the mean action, and the third term is a KL-divergence penalty.”
BackgroundMethodInfluentialSelf-citationA Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941
Neural networks for control systems—A survey
Hunt, K. J., Sbarbaro, D. G., Żbikowski, R., & Gawthrop, P. J. (1992) · Automatica · doi:10.1016/0005-1098(92)90053-i · Cited by 1,874 · cited 1×
“neural networks have been used for control for decades [12], [33], limitations on computation and algorithms have made large neural network policies very difficult to learn.”
MethodRobust model predictive control of constrained linear systems with bounded disturbances
Mayne, D. Q., Seron, M. M., & Raković, S. V. (2004) · Automatica · doi:10.1016/j.automatica.2004.08.019 · Cited by 1,680 · cited 1×
“Model predictive control (MPC) is an effective and reliable method for controlling robotic systems, particularly autonomous aerial vehicles such as quadcopters, because of its robustness to moderate model errors [26], ability to use high-level objectives [41], and relative simplicity.”
MethodALVINN, an autonomous land vehicle in a neural network
Pomerleau, D. (2018) · doi:10.1184/r1/6603146 · Cited by 1,369
Flagged first, then by citations
2 of the 45 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Items 401–445 of 445
Optimization, Robustness and Risk-Sensitivity in Machine Learning: A Dynamical Systems Perspective
Nar, K. (2020) · eScholarship (California Digital Library)
Interpolation for Parametric ROMs
Brunton, S. L., & Kutz, J. N. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108380690.013
Robotic Visuomotor Control with Unsupervised Forward Model Learned from Videos.Preprint
Yuan, H., Wu, R., Zhao, A., Zhang, H., Ding, Z., & Dong, H. (2021) · arXiv (Cornell University)
Reinforcement Learning using Guided ObservabilityPreprint
Weigand, S., Klink, P., Peters, J., & Pajarinen, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.10986
“Other methods are either application specific [12], utilize models [13], [14], or make the policy assume information which is not available during online operation [15], [16].”
MethodKahn, G. (2020) · eScholarship (California Digital Library)
Self-citationLearning Stable Adaptive Explicit Differentiable Predictive Control for Unknown Linear SystemsPreprint
Drgoňa, J., Tuor, A. R., & Vrabie, D. (2020) · arXiv (Cornell University)
Online-Learning Deep Neuro-Adaptive Dynamic Inversion Controller for Model Free ControlPreprint
Lutes, N. A., Krishnamurthy, K., Nadendla, V. S. S., & Balakrishnan, S. N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.10383
“In [13], model predictive control is used to train a DRL controller for control of a quadcopter drone.”
Deep Reinforcement Learning Survey from the Peopective of Action Planning
Ariki, Y. (2021) · Journal of the Robotics Society of Japan · doi:10.7210/jrsj.39.601
Optimal Control-Based Baseline for Guided Exploration in Policy Gradient MethodsPreprint
Lyu, X., Li, S., Siriya, S., Pu, Y., & Chen, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.02073
“In [24], [25], model predictive control (MPC) has been employed to either generate guided training data or provide trajectory optimizationbased supervision for RL to learn complex, high-dimensional policies.”
Methodde Morais, G. A. P. (2020) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.18.2020.tde-25082020-092618
Prediction Error-Based Action Policy Learning for Quadcopter Flight Control
Khanzada, J. S., Muhammad, W., & Irshad, M. J. (2021) · International Conference on Energy, Power and Environment · doi:10.3390/engproc2021012047
(2019) · Cambridge University Press eBooks · doi:10.1017/9781108380690.015
Brunton, S. L., & Kutz, J. N. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108380690.001 · Cited by 1
Brunton, S. L., & Kutz, J. N. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108380690.014
Brunton, S. L., & Kutz, J. N. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108380690.016
Lin, X., Chen, Q., Zhou, B., & Peng, Y. (2022) · Research Square · doi:10.21203/rs.3.rs-1802712/v1
Multi-aircraft Attack Guidance Law Based on Supervised Learning
Lyu, Z., Guo, X., Xu, C. L., Hao, M., & Liang, X. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-0479-2_253
Kshirsagar, A., Faibish, T., Hoffman, G., & Biess, A. (2022) · International Conference on Robotics, Automation and Artificial Intelligence (RAAI) · doi:10.1109/raai56146.2022.10092989
“Since then researchers have proposed several variations of the GPS algorithm [4], and have successfully utilized them for autonomous manipulation [5]–[8], and locomotion tasks [1], [2], [6], [9].”
MethodNémeth, B., & Gáspár, P. (2023) · Advances in industrial control · doi:10.1007/978-3-031-30537-5_1
SpeedFormer: Learning Speed Profiles with Upper and Lower Boundary Constraints Based on Transformer
Yeon, K., Kim, H., & Jeong, S.-G. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341689
“In [9], an MPC-guided policy search algorithm has been proposed, where the MPC is used to generate data at training time under full state observations provided by an instrumented training environment.”
MethodMagesh, M., Jawahar, P. K., Saranya, S. N., & Raj Jawahar, R. (2024) · Elektronika ir Elektrotechnika · doi:10.5755/j02.eie.34343
“The SMO-MPC was found to be robust to model errors, to update predictions based on measurable data during runtime, making it more resilient to model errors, and enhancing landing performance in unpredictable dynamics [20].”
BackgroundPreliminary Study about Desalinated Water and Alkalinity
Benouali, D., Benhammadi, S., KHERICI, S., & Kadri, S. (2024) · International Journal of Scientific Research and Management (IJSRM) · doi:10.18535/ijsrm/v12i04.c01
A Survey on Theories and Application for Self Driving Cars on Deep Learning Methods
Verma, P., Chauhan, L., & Srivastava, P. A. K. (2024) · International Journal of Scientific Research and Management (IJSRM) · doi:10.18535/ijsrm/v12i04.ec06
Hoover, R. J., Wu, W., & Shimada, K. (2024) · International Conference on Automation, Robotics and Applications · doi:10.1109/icara60736.2024.10553186
“A network was trained using the state information control actions provided by a Model Predictive Control (MPC) algorithm [12].”
MethodLi, L., Pei, W., Zhang, Y., & Ma, B. (2024) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2024.11.145
Zhu, L., Miao, J., Chen, C., Yan, B., Gong, T., Yu, F. R., & Tang, T. (2025) · Computers & Industrial Engineering · doi:10.1016/j.cie.2025.111372
Approximate Robust Tube Nonlinear Model Predictive Control for Vehicle Collision Avoidance
Kim, S., Han, K., & Choi, S. B. (2025) · IEEE Conference on Control Technology and Applications (CCTA) · doi:10.1109/ccta53793.2025.11151526
“Existing studies have applied IL to NMPC in quadrupedal robot’s gait control [9], drone control [10], and drift control using remote control vehicles [11], proving its effectiveness in maintaining the NMPC’s great control performance and reducing the computation load by the amount of 10-20 times.”
Learning Robust Agile Flight Control with Stability Guarantees
Pries, L., & Ryll, M. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11246103
“Several works train a policy to map state observations directly to desired individual rotor thrusts [31]–[33].”
Yin, F., Watson, D., & Lam, H. (2025) · International Conference on Advanced Robotics and Mechatronics (ICARM) · doi:10.1109/icarm65671.2025.11293635
“Recurrent neural networks (RNNs) have been integrated into bipedal robot controllers [5] to approximate static and dynamic friction, while deep neural networks (DNNs) serve as model predictive controllers for quadcopters [6], estimating their position from onboard sensors.”
A Guided Retraining Strategy For Safe ReLU Neural Network Controllers Of Linear Systems
Zago, J. G., Notarstefano, G., & Camponogara, E. (2025) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc57313.2025.11312068
“If R k ∩ O ̸ = /0, two main strategies have been proposed in the literature to modify the neural network and attain the desired property: i) directly modify the weights ( W l , θ l ) of the neural network, ∀ l ∈ { 1 ,... , L } [22]; ii) improve the training dataset of the neural network [15].”
Engineering Safe Machine Learning for Automated Driving Systems
Frtunikj, J., & Fürst, S. (2019) · Safety-Critical Systems Symposium (SSS) · doi:10.65391/r1070
Advanced LPV-MPC for UAV Control Using Particle Swarm Optimization
Hamza, A. M., & Asma, M. (2025) · International Conference on Research in Computing at Feminine (RIF) · doi:10.1109/rif68108.2025.11406840
“To overcome such limitations, more advanced methods have emerged, including Linear Parameter-Varying (LPV) modelling [9], reinforcement learning based control [10], and MPC-guided deep learning frameworks [11].”
Micro-Data Reinforcement Learning for Adaptive Robots
Chatzilygeroudis, K. I. (2018) · theses.fr (ABES) · doi:10.70675/7990d9edz7f40z4344z9318z6da4e46000c0
“This can be realized by using MPC with full state information in some training phase, to learn neural network policies that take the raw observations as input, thus, not requiring full state information at test time (Zhang et al., 2016).”
MethodIdentification de systèmes par réseaux de neurones pour la commande prédictive.
Mahé, A. (2020) · theses.fr (ABES) · doi:10.70675/c916113dz6bbaz4db6z8873z099dd168cba8
Patel, N., Evangeliou, N., Krishnamurthy, P., Tzes, A., & Khorrami, F. (2026) · Autonomous Robots · doi:10.1007/s10514-026-10250-x
Peng, C., Pan, Y., Yang, X., Wang, N., Yu, Z., & Li, G. (2026) · IEEE Journal of Oceanic Engineering · doi:10.1109/joe.2026.3695320
“MPC has been widely applied in robotics for tasks involving nonlinear dynamics and complex constraints, such as quadrotor and humanoid control [30], [31], [32].”
Ma, Y., Zou, Y., Wu, Y., He, L., Zhang, K., & Wang, D. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.115923
Hirst, C. A., Reale, C., & Frew, E. W. (2026) · Journal of Guidance Control and Dynamics · doi:10.2514/1.g009648
A comprehensive review of AI-based UAV navigation techniques in GPS-denied environments
Kalaimani, N., Saran, R., & Vignesh, G. (2026) · International Journal of Intelligent Unmanned Systems · doi:10.1108/ijius-03-2026-0077
Look as You Leap: Planning Simultaneous Motion and Perception for High-DoF Robots
Meng, Q., Flores, E., Quintero-Peña, C., Qian, P., Kingston, Z., Hamlin, S. K., Unhelkar, V. V., & Kavraki, L. E. (2026) · IEEE Transactions on Robotics · doi:10.1109/tro.2026.3731487
“Thepolicylearnerhas accesstoprivilegedinformationduringtraining,e.g.,byusing MPC[48],aplanner[18],orastate-basedtrainedexpert[19].”
Benchmarking Model-Based Reinforcement Learning
Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., Zhang, S., Zhang, G., Abbeel, P., & Ba, J. (2019) · arXiv · doi:10.48550/arxiv.1907.02057 · Cited by 226
Self-citationTransferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
James, S., Davison, A. J., & Johns, E. (2017) · arXiv · doi:10.48550/arxiv.1707.02267 · Cited by 208
“A slightly different approach is guided policy search (GPS) [9, 8], which in addition to classic control benchmarks, has achieved success in flight [24, 7], and robot manipulation [10, 14, 13].”
Background
Metadata
- Authors
- Zhang, T., Kahn, G., Levine, S., & Abbeel, P.
- Affiliations
- University of California, Berkeley (US)
- Venue
- 2016 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2016-05
- Pages
- 528-535
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W1923344279
- Cited by
- 452
- References
- 45
- Citations by year
- 2026: 10 · 2025: 18 · 2024: 37 · 2023: 33 · 2022: 54 · 2021: 53 · 2020: 65 · 2019: 93 · 2018: 42 · 2017: 39 · 2016: 8
- FWCI
- 32.08
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (N66001)
- Army Research Office
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Physical Sciences › Engineering › Control and Systems Engineering › Advanced Control Systems Optimization
- Keywords
- model predictive control, deep neural network policies, autonomous aerial vehicles, quadrotor control, obstacle avoidance, reinforcement learning, state estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
