The DOI matches its Crossref record; 43 of the 45 references it cites were found in a registry or an index.
Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search
Open accessAbstract
Model predictive control (MPC) is an effective method for controlling robotic systems, particularly autonomous aerial vehicles such as quadcopters. However, application of MPC can be computationally demanding, and typically requires estimating the state of the system, which can be challenging in complex, unstructured environments. Reinforcement learning can in principle forego the need for explicit state estimation and acquire a policy that directly maps sensor readings to actions, but is difficult to apply to unstable systems that are liable to fail catastrophically during training before an effective policy has been found. We propose to combine MPC with reinforcement learning in the framework of guided policy search, where MPC is used to generate data at training time, under full state observations provided by an instrumented training environment. This data is used to train a deep neural network policy, which is allowed to access only the raw observations from the vehicle's onboard sensors. After training, the neural network policy can successfully control the robot without knowledge of the full state, and at a fraction of the computational cost of MPC. We evaluate our method by learning obstacle avoidance policies for a simulated quadrotor, using simulated onboard sensors and no explicit state estimation at test time.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The frequent sensing and re-planning allow the robot to react dynamically to changes in the environment, while still making more intelligent choices than a purely reactive system (Erez et al., 2011; Lenz et al., 2015; Zhang et al., 2016; Liu et al., 2017; Williams et al., 2017; Gillespie et al., 2018; Amos et al., 2019; Kabzan et al., 2019).”
From Lessons for Robotics From the Control Architecture of the Octopus (2022) · cites this paper as Zhang et al., 2016
In the citing paper
“AField-Programmable Gate Array (FPGA) is another viable solution, showing great improvements in both performance and power consumption in deep learning applications.”
From A survey of deep learning techniques for autonomous driving (2020) · cites this paper as [170]
In the citing paper
“Currently, the largest portion of papers in deep learning for selfdriving cars are focused mainly on perception and End2End learning [81, 124].”
From A survey of deep learning techniques for autonomous driving (2020) · cites this paper as [124]
In the citing paper
“Finally, realtime requirements have to be fulfilled for processing the large amounts of data gathered from the car's sensors suite, as well as for updating the parameters of deep learning systems over high-speed communication lines [170].”
From A survey of deep learning techniques for autonomous driving (2020) · cites this paper as [170]
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“We used default values for all learning parameters as presented in [14].”
Method(2010) · International Conference on Machine Learning · Cited by 13,995
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · cited 1×
“More recently, deep neural network policies have been used for tasks ranging from robotic control [22], [23] to video game playing [27].”
MethodMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · cited 2×
“However, our derivation largely follows prior work [41].”
MethodResultReinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160 · cited 1×
“An overview of recent reinforcement learning methods in robotics can be found in a recent survey paper [15].”
BackgroundEnd-to-End Training of Deep Visuomotor PoliciesPreprint
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 2,641 · cited 10×
“In this paper, we use the BADMM version, which specifies the following objective for trajectory optimization: min pi(τ) T∑ t=1 Epi(xt,ut) [ `(xt,ut)− uTt λiµt + νitDKL ( pi(ut|xt)‖πθ(ut|xt) )] (1) where λµt is a Lagrange multiplier on the mean action, and the third term is a KL-divergence penalty.”
BackgroundMethodInfluentialSelf-citationA Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941
Neural networks for control systems—A survey
Hunt, K. J., Sbarbaro, D. G., Żbikowski, R., & Gawthrop, P. J. (1992) · Automatica · doi:10.1016/0005-1098(92)90053-i · Cited by 1,874 · cited 1×
“neural networks have been used for control for decades [12], [33], limitations on computation and algorithms have made large neural network policies very difficult to learn.”
MethodRobust model predictive control of constrained linear systems with bounded disturbances
Mayne, D. Q., Seron, M. M., & Raković, S. V. (2004) · Automatica · doi:10.1016/j.automatica.2004.08.019 · Cited by 1,680 · cited 1×
“Model predictive control (MPC) is an effective and reliable method for controlling robotic systems, particularly autonomous aerial vehicles such as quadcopters, because of its robustness to moderate model errors [26], ability to use high-level objectives [41], and relative simplicity.”
MethodALVINN, an autonomous land vehicle in a neural network
Pomerleau, D. (2018) · doi:10.1184/r1/6603146 · Cited by 1,369
Cited by
Sorted by citations
Efficient Processing of Deep Neural Networks: A Tutorial and Survey
Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2761740 · Cited by 4,215
“• Robotics DNNs have been successful in the domain of robotic tasks such as grasping with a robotic arm [32], motion planning for ground robots [33], visual navigation [4, 34], control to stabilize a quadcopter [35] and driving strategies for autonomous vehicles [36].”
A survey of deep learning techniques for autonomous driving
Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) · Journal of Field Robotics · doi:10.1002/rob.21918 · Cited by 1,809
“rently, the largest portion of papers in deep learning for selfdriving cars are focused mainly on perception and End2End learning [81, 124].”
BackgroundInfluentialData-Driven Science and Engineering
Brunton, S. L., & Kutz, J. N. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108380690 · Cited by 1,269
Socially aware motion planning with deep reinforcement learning
Chen, Y. F., Everett, M., Liu, M., & How, J. P. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202312 · Cited by 743
“While several recent works have applied deep RL to motion planning [22], [23], they are mainly focused on single agent navigation in unknown static environments, and with an emphasis on computing control inputs directly from raw”
BackgroundDecentralized non-communicating multiagent collision avoidance with deep reinforcement learning
Chen, Y. F., Liu, M., Everett, M., & How, J. P. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989037 · Cited by 689
“While applications of RL are typically limited to discrete, low-dimensional domains, recent advances in Deep RL [20]– [23] have demonstrated human-level performance in complex, high-dimensional spaces.”
BackgroundControl of a Quadrotor With Reinforcement Learning
Hwangbo, J., Sa, I., Siegwart, R., & Hutter, M. (2017) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2017.2720851 · Cited by 634
“Guided policy search with a MPC controller [5] is demonstrated in simulation.”
MethodSparse identification of nonlinear dynamics for model predictive control in the low-data limit
Kaiser, E., Kutz, J. N., & Brunton, S. L. (2018) · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rspa.2018.0335 · Cited by 657
“Nonlinear models based on machine learning, such as neural networks, are increasingly used due to advances in computing power, and recently deep reinforcement learning has been combined with MPC [36, 56], yielding impressive results in the large-data limit.”
Model-Based Reinforcement Learning for Closed-Loop Dynamic Control of Soft Robotic Manipulators
Thuruthel, T. G., Falotico, E., Renda, F., & Laschi, C. (2018) · IEEE Transactions on Robotics · doi:10.1109/tro.2018.2878318 · Cited by 409
“The state-of-the-art approaches using variations of this idea are concerned with using learned local models [20], compound multistep controllers [21], and deep representation of the control policies [22].”
MethodData-driven discovery of Koopman eigenfunctions for control
Kaiser, E., Kutz, J. N., & Brunton, S. L. (2021) · Machine Learning Science and Technology · doi:10.1088/2632-2153/abf0f5 · Cited by 353
Fan, T., Long, P., Liu, W., & Pan, J. (2020) · The International Journal of Robotics Research · doi:10.1177/0278364920916531 · Cited by 337
“DNNs have also been integrated with model predictive control to control robotic systems (Lenz et al., 2015; Zhang et al., 2016b).”
MethodDeep learning in robotics: A review of recent research
Pierson, H. A., & Gashler, M. S. (2017) · Advanced Robotics · doi:10.1080/01691864.2017.1365009 · Cited by 331
“Levine et al. [128] used reinforcement learning (Structure D) to show that this can be superior to separate systems for perception and control.”
BackgroundInfluentialFasel, U., Kutz, J. N., Brunton, B. W., & Brunton, S. L. (2022) · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rspa.2021.0904 · Cited by 359
“Deep neural network models have been increasingly used for deep MPC [83, 84]; however, they often rely on access to massive data sets, have limited ability to generalize, do not readily incorporate known physical constraints, and are computationally expensive.”
MethodEnergy Management Strategy for a Hybrid Electric Vehicle Based on Deep Reinforcement Learning
Hu, Y., Li, W., Xu, K., Zahid, T., Qin, F., & Li, C. (2018) · Applied Sciences · doi:10.3390/app8020187 · Cited by 281
“The experimental results are given in Section 4, followed by the conclusions in Section 5.”
Learning an Approximate Model Predictive Controller With Guarantees
Hertneck, M., Köhler, J., Trimpe, S., & Allgöwer, F. (2018) · IEEE Control Systems Letters · doi:10.1109/lcsys.2018.2843682 · Cited by 278
“[5, 6, 7] can be adapted similar to the proposed framework, if a robust control method is used in combination with the proposed validation method.”
BackgroundMethodApproximate model predictive building control via machine learning
Drgoňa, J., Picard, D., Kvasnica, M., & Helsen, L. (2018) · Applied Energy · doi:10.1016/j.apenergy.2018.02.156 · Cited by 260
Motion planning and control for mobile robot navigation using machine learning: a survey
Xiao, X., Liu, B., Warnell, G., & Stone, P. (2022) · Autonomous Robots · doi:10.1007/s10514-022-10039-8 · Cited by 320
“[111], who addressed the problem of partial observability of UAV obstacle avoidance by training a system with access to full state information and deployment with only partial observations.”
BackgroundMethodInfluentialVulnerability of Deep Reinforcement Learning to Policy Induction Attacks
Behzadan, V., & Munir, A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-62416-7_19 · Cited by 247
“…both feature selection and Q-function approximation, hence enabling unprecedented performance in complex settings such as learning efficient playing strategies from unlabeled video frames of Atari games [7], robotic manipulation [8], and autonomous navigation of aerial [9] and ground vehicles [10].”
BackgroundData-driven science and engineering : machine learning, dynamical systems, and control
Brunton, S. L., & Kutz, J. N. (2019) · doi:10.1017/978-1-009-08951-7 · Cited by 270
Nonlinear and Adaptive Intelligent Control Techniques for Quadrotor UAV – A Survey
Mo, H., & Farid, G. (2018) · Asian Journal of Control · doi:10.1002/asjc.1758 · Cited by 244
“Reinforcement learning based studies to train a quadrotor fl ight controller for re fi ned and precise maneuverers, have also been reported in the literature [133 – 135].”
MethodResultHu, X., Liu, T., Qi, X., & Barth, M. (2019) · IEEE Industrial Electronics Magazine · doi:10.1109/mie.2019.2913015 · Cited by 240
“The upper level used MPC to determine the velocity trajectories to avoid red-light idling, and the lower level applied RL to optimize the energy-efficient controls for each HEV [36].”
MethodDi, X. S., & Shi, R. (2021) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2021.103008 · Cited by 256
A Deep Reinforcement Learning Strategy for UAV Autonomous Landing on a Moving Platform
Rodríguez-Ramos, A., Sampedro, C., Bavle, H., de la Puente, P., & Campoy, P. (2018) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-018-0891-8 · Cited by 229
“Nevertheless, some recent advances have proposed novel methods to learn low-level control policies from imperfect sensor data in simulation [49,21].”
BackgroundMethodSmolyanskiy, N., Kamenev, A., Smith, J., & Birchfield, S. T. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206285 · Cited by 230
“[33] use DRL to learn obstacle avoidance policies for a simulated quadrotor.”
MethodDeep Reinforcement Learning Attitude Control of Fixed-Wing UAVs Using Proximal Policy optimization
Bøhn, E., Coates, E. M., Moe, S., & Johansen, T. A. (2019) · International Conference on Unmanned Aircraft Systems (ICUAS) · doi:10.1109/icuas.2019.8798254 · Cited by 219
“In [33], the authors use a model predictive controller (MPC) to generate training data for an RL controller, thereby guiding the policy search and avoiding the potentially catastrophic early phase before an effective policy is found.”
MethodLearning an approximate model predictive controller with guarantees
Hertneck, M. (2018) · OPUS Publication Server of the University of Stuttgart (University of Stuttgart) · doi:10.18419/opus-9722 · Cited by 213
Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assembly
Luo, J., Solowjow, E., Wen, C., Ojea, J. A., Agogino, A. M., Tamar, A., & Abbeel, P. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793506 · Cited by 195
“b) Related Work: Recent advances in RL have gained great success in solving a variety of problems from playing video games [9, 10, 11] to robotic locomotion [12, 13, 14, 15, 16], manipulation [17, 18, 19, 20, 4, 21, 22, 23, 24, 25].”
BackgroundSelf-citationBaccour, E., Mhaisen, N., Abdellatif, A. A., Erbad, A., Mohamed, A. M. S., Hamdi, M., & Guizani, M. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3200740 · Cited by 223
“online learning is broadly exploited for UAVs/robots guidance, including the work in [58] where drones learn how to navigate and avoid obstacles while searching target objects.”
BackgroundDeterministic Policy Gradient With Integral Compensator for Robust Quadrotor Control
Wang, Y., Sun, J. L., He, H., & Sun, C. (2019) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2018.2884725 · Cited by 180
“In[27], an obstacle avoidance policy for quadrotors trained by the guided policy search with MPC is demonstrated in simulations.”
MethodTemporal Difference Models: Model-Free Deep RL for Model-Based ControlPreprint
Pong, V. H., Gu, S., Dalal, M., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09081 · Cited by 224
“…function approximators such as deep neural networks, RL can provide impressive results on tasks ranging from playing games (Mnih et al., 2015; Silver et al., 2016), to flying and driving (Lillicrap et al., 2015; Zhang et al., 2016), to controlling robotic arms (Levine et al., 2016; Gu et al., 2017).”
BackgroundSelf-citationKaufmann, E., Loquercio, A., Ranftl, R., Müller, M., Koltun, V., & Scaramuzza, D. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.040 · Cited by 162
Learning nonlinear dynamic models of soft robots for model predictive control with neural networks
Gillespie, M. T., Best, C. M., Townsend, E. C., Wingate, D., & Killpack, M. D. (2018) · IEEE International Conference on Soft Robotics (RoboSoft) · doi:10.1109/robosoft.2018.8404894 · Cited by 156
“For the work in [21], they used a DNN to learn a MPC control policy for UAVs.”
MethodState-of-the-Art Intelligent Flight Control Systems in Unmanned Aerial Vehicles
Santoso, F., Garratt, M., & Anavatti, S. G. (2017) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2017.2651109 · Cited by 146
“[95] demonstrate that through the guided policy search, achieved by decomposing the search policy into a trajectory optimization phase, effective memorization and recall techniques can be achieved.”
BackgroundMethodLearning a deep neural net policy for end-to-end control of autonomous vehicles
Rausch, V., Hansen, A., Solowjow, E., Liu, C., Kreuzer, E. J., & Hedrick, J. K. (2017) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc.2017.7963716 · Cited by 140
“Since explicit mapping of observations to control actions has not to be determined a priori in a design process anymore, deep learning turns out to be very efficient for certain high level controls objectives [6].”
BackgroundMultistep Prediction of Dynamic Systems With Recurrent Neural Networks
Mohajerin, N., & Waslander, S. L. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2891257 · Cited by 150
“In [48], a deep NN, trained by an MPC policy search, is used as a policy learner to control a quadrotor.”
MethodLiu, Y., He, X., Liu, D., & Wang, L. (2022) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2022.102365 · Cited by 139
A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials
Chatzilygeroudis, K., Vassiliades, V., Stulp, F., Calinon, S., & Mouret, J.-B. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2019.2958211 · Cited by 160
“This can be realized by using MPC with full state information in some training phase, to learn neural network (NN) policies that take the raw observations as input, thus, not requiring full state information at test time [195].”
MethodPolicy Search for Model Predictive Control With Application to Agile Drone Flight
Song, Y., & Scaramuzza, D. (2022) · IEEE Transactions on Robotics · doi:10.1109/tro.2022.3141602 · Cited by 131
Efficient Processing of Deep Neural Networks
Sze, V., Chen, Y., Yang, T.-J., & Emer, J. (2020) · Synthesis lectures on computer architecture · doi:10.2200/s01004ed1v01y202004cac050 · Cited by 125
Safe deep reinforcement learning-based adaptive control for USV interception mission
Du, B., Lin, B., Zhang, C., Dong, B., & Zhang, W. (2022) · Ocean Engineering · doi:10.1016/j.oceaneng.2021.110477 · Cited by 110
PLATO: Policy learning using adaptive trajectory optimization
Kahn, G., Zhang, T., Levine, S., & Abbeel, P. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989379 · Cited by 123
“The motivation behind PLATO is most similar to the MPC variant of guided policy search (MPC-GPS) [12].”
MethodResultInfluentialSelf-citationDeep Reinforcement Learning for Robotic Assembly of Mixed Deformable and Rigid Objects
Luo, J., Solowjow, E., Wen, C., Ojea, J. A., & Agogino, A. M. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594353 · Cited by 105
“Recent advances in reinforcement learning have gained great success in solving a variety of problems from playing video games [5, 6] to robotic locomotion [7, 8, 9, 10], and manipulation [11, 12, 13, 14, 15].”
BackgroundA Survey on Design, Actuation, Modeling, and Control of Continuum Robot
Zhang, J., Fang, Q., Xiang, P., Sun, D., Xue, Y., Jin, R., Qiu, K., Xiong, R., Wang, Y., & Lu, H. (2022) · Cyborg and Bionic Systems · doi:10.34133/2022/9754697 · Cited by 104
“The most advanced methods of using this variant of the idea involve the use of local models of learning [126], composite multistep controllers [127], and deep representations of control strategies [128].”
MethodNeural Networks and Deep Learning
Brunton, S. L., & Kutz, J. N. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108380690.007 · Cited by 105
Near-Optimal Rapid MPC Using Neural Networks: A Primal-Dual Policy Learning Framework
Zhang, X., Bujarbaruah, M., & Borrelli, F. (2020) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2020.3024571 · Cited by 93
Neural network-based flight control systems: Present and future
Emami, S. A., Castaldi, P., & Banazadeh, A. (2022) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2022.04.006 · Cited by 96
Autonomous Navigation for eVTOL: Review and Future Perspectives
Wei, H., Lou, B., Zhang, Z., Liang, B., Wang, F., & Lv, C. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3352613 · Cited by 93
Review of intelligent energy management techniques for hybrid electric vehicles
Urooj, A., & Nasir, A. (2024) · Journal of Energy Storage · doi:10.1016/j.est.2024.112132 · Cited by 97
Flying Free: A Research Overview of Deep Learning in Drone Navigation Autonomy
Lee, T., McKeever, S., & Courtney, J. (2021) · Drones · doi:10.3390/drones5020052 · Cited by 92
Yang, X., & Wei, P. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3048360 · Cited by 86
Xie, L., Wang, S., Rosa, S., Markham, A., & Trigoni, N. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8461203 · Cited by 83
A Benchmark Comparison of Learned Control Policies for Agile Quadrotor Flight
Kaufmann, E., Bauersfeld, L., & Scaramuzza, D. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811564 · Cited by 86
Guided Policy Search via Approximate Mirror DescentPreprint
Montgomery, W. H., & Levine, S. (2016) · arXiv (Cornell University) · Cited by 82
Self-citationDeep Visual MPC-Policy Learning for Navigation
Hirose, N., Xia, F., Martín-Martín, R., Sadeghian, A., & Savarese, S. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2925731 · Cited by 90
SINDy with Control: A Tutorial
Fasel, U., Kaiser, E., Kutz, J. N., Brunton, B. W., & Brunton, S. L. (2021) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc45484.2021.9683120 · Cited by 81
Model Predictive Control for Micro Aerial Vehicles: A Survey
Nguyen, H., Kamel, M., Alexis, K., & Siegwart, R. (2021) · European Control Conference (ECC) · doi:10.23919/ecc54610.2021.9654841 · Cited by 80
Hu, H., Fazlyab, M., Morari, M., & Pappas, G. J. (2020) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc42340.2020.9304296 · Cited by 89
“In autonomous systems, DNNs are either used as feedback controllers [1], [2], motion planners [3], perception modules, or end-to-end controllers [4].”
Drgoňa, J., Kiš, K., Tuor, A. R., Vrabie, D., & Klaučo, M. (2022) · Journal of Process Control · doi:10.1016/j.jprocont.2022.06.001 · Cited by 77
“These approaches are known in the machine learning community as imitation learning [33], or MPC-guided policy search [34].”
BackgroundInternet of drones security: Taxonomies, open issues, and future directions
Derhab, A., Cheikhrouhou, O., Allouch, A., Koubâa, A., Qureshi, B., Ferrag, M. A., Maglaras, L. A., & Khan, F. A. (2022) · Vehicular Communications · doi:10.1016/j.vehcom.2022.100552 · Cited by 77
Learning control lyapunov functions from counterexamples and demonstrations
Ravanbakhsh, H., & Sankaranarayanan, S. (2018) · Autonomous Robots · doi:10.1007/s10514-018-9791-9 · Cited by 74
“On one hand, these are much less powerful than approaches that use neural networks [105], for instance.”
BackgroundMethodLearning quadrotor dynamics for precise, safe, and agile flight control
Saviolo, A., & Loianno, G. (2023) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2023.03.009 · Cited by 79
Recchiuto, C. T., & Sgorbissa, A. (2017) · Journal of Field Robotics · doi:10.1002/rob.21756 · Cited by 78
Policy Learning for Nonlinear Model Predictive Control With Application to USVs
Wang, R., Li, H., Liang, B., Shi, Y., & Xu, D. (2023) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2023.3274869 · Cited by 71
“The work in [18] used MPC to guide reinforcement learning (RL) training to improve the training efficiency of RL and guarantee the safety of the RL results.”
MethodConstrained Cross-Entropy Method for Safe Reinforcement Learning
Wen, M., & Topcu, U. (2020) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2020.3015931 · Cited by 75
“By encoding control tasks as reward or cost functions, RL has successfully solved a wide range of tasks such as Atari games [19; 20], the game of Go [31; 32], controlling simulated robots [36; 24] and real robots [15; 37; 22].”
BackgroundModel-Based Control of Soft Actuators Using Learned Non-linear Discrete-Time Models
Hyatt, P., Wingate, D., & Killpack, M. D. (2019) · Frontiers in Robotics and AI · doi:10.3389/frobt.2019.00022 · Cited by 74
Greatwood, C. M., & Richards, A. G. (2019) · Autonomous Robots · doi:10.1007/s10514-019-09829-4 · Cited by 70
“Zhang et al. (2016) used MPC as the supervisor for their learning algorithm, resulting in a deep neural network policy for obstacle avoidance, while Aswani et al. (2013) used a learning ‘oracle’ to refine the prediction model for a general MPC. Sharma and Taylor (2012) used RL for waypoint…”
MethodHybrid Reinforcement Learning Control for a Micro Quadrotor Flight
Yoo, J., Jang, D., Kim, H. J., & Johansson, K. H. (2020) · IEEE Control Systems Letters · doi:10.1109/lcsys.2020.3001663 · Cited by 67
“On the other hand, similar to some RL meth-ods to use a system model [6]–[8], LQR-RL can be used if a system model is given, where LQR is an optimal control method.”
MethodMarshall, J. A., Sun, W., & L’Afflitto, A. (2021) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2021.10.013 · Cited by 71
“Alternative approaches to the outer-loop design process require to find 𝜙 ( ⋅ ) and 𝜃 ( ⋅ ) such that the position in vehicle’s horizontal plane is regulated (Adıgüzel & Mumcu, 2019; Hamrah & Sanyal, 2020; Xiong & Zhang, 2016).”
BackgroundMethodIslam, M., Okasha, M., & Sulaeman, E. (2019) · International Journal of Control Automation and Systems · doi:10.1007/s12555-018-0860-9 · Cited by 66
Deep Learning in Diverse Intelligent Sensor Based Systems
Zhu, Y., Wang, M., Yin, X., Zhang, J., Meijering, E. H. W., & Hu, J. (2022) · Sensors · doi:10.3390/s23010062 · Cited by 73
Learning Perception-Aware Agile Flight in Cluttered Environments
Song, Y., Shi, K., Pěnička, R., & Scaramuzza, D. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160563 · Cited by 75
“Some other works propose to learn end-to-end policies directly from sensory observations to control commands using imitation learning [20]–[22].”
BackgroundA Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to ImitationPreprint
Tai, L., Zhang, J., Liu, M., Boedecker, J., & Burgard, W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.07139 · Cited by 111
“Most of the reinforcement learning applications in robotics are trained in simulation environments (Tai and Liu, 2016a; Zhang et al., 2016a).”
BackgroundMethodInfluentialMiao, Z., Li, Y., & Wang, X. (2021) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3448016.3457258 · Cited by 65
Data-driven planning via imitation learning
Choudhury, S., Bhardwaj, M., Arora, S., Kapoor, A., Ranade, G. V., Scherer, S., & Dey, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364918781001 · Cited by 70
“mitation of clairvoyant oracles, that has access to more information than the learner during training, to improve reinforcement learning as they offer better sample efficiency and safety. Zhang et al. [130], Kahn et al. [57] train policies that map current observation to action by extending guided policy search [73] fo…”
BackgroundToward autonomous mapping and exploration for mobile robots through deep supervised learning
Bai, S., Chen, F., & Englot, B. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206050 · Cited by 58
“challenging problems such as image recognition [17], robot manipulation [18] and control [19], [20].”
BackgroundDeep-learning in Mobile Robotics - from Perception to Control Systems: A Survey on Why and Why not.Preprint
Tai, L., & Liu, M. (2016) · arXiv (Cornell University) · Cited by 59
Car-Following Models for Human-Driven Vehicles and Autonomous Vehicles: A Systematic Review
Wang, Z., Shi, Y., Tong, W., Gu, Z., & Cheng, Q. (2023) · Journal of Transportation Engineering Part A Systems · doi:10.1061/jtepbs.teeng-7836 · Cited by 65
UAV Obstacle Avoidance Algorithm to Navigate in Dynamic Building Environments
Aldao, E., González-deSantos, L. M., Michinel, H., & González-Jorge, H. (2022) · Drones · doi:10.3390/drones6010016 · Cited by 56
“Some of the most common are based on geometric relations [18,19], fuzzy logic [20,21], potential fields [22,23], and neural networks [24–26], among others.”
BackgroundNeural network modeling for steering control of an autonomous vehicle
Garimella, G., Funke, J., Wang, C., & Kobilarov, M. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206084 · Cited by 56
Vision-Based High-Speed Driving With a Deep Dynamic Observer
Drews, P., Williams, G., Goldfain, B., Theodorou, E. A., & Rehg, J. M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2896449 · Cited by 54
Task-Oriented Deep Reinforcement Learning for Robotic Skill Acquisition and Control
Xiang, G., & Su, J. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2949596 · Cited by 56
“…been extensively studied on robotic skill acquisition and control problems, such as robot locomotion [7], [8]; sensorimotor control [9], [10]; robotic manipulation [11], [12]; dextrous robot grasping [13], [14]; self-driving vehicles [15], [16]; and autonomous helicopters [17], [18], to name a few.”
BackgroundXiao, C., Lu, P., & He, Q. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3107742 · Cited by 58
“[3] applied guided policy search (GPS) to a quadrotor collision avoidance task.”
MethodXiong, H., Ma, T., Zhang, L., & Diao, X. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.10.020 · Cited by 54
OmniDrones: An Efficient and Flexible Platform for Reinforcement Learning in Drone Control
Xu, B., Gao, F., Yu, C., Zhang, R., Wu, Y., & Wang, Y. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3356168 · Cited by 75
“Prior works explored end-to-end training of visual-motor control policies [2], [24], [25], [26] to avoid the need for explicit dynamics modeling and hand-engineered control stack.”
BackgroundSafe and Near-Optimal Policy Learning for Model Predictive Control using Primal-Dual Neural Networks
Zhang, X., Bujarbaruah, M., & Borrelli, F. (2019) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc.2019.8814335 · Cited by 53
Memory Augmented Control NetworksPreprint
Khan, A., Zhang, C., Atanasov, N., Karydis, K., Kumar, V., & Lee, D. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.05706 · Cited by 53
“Purely convolutional architectures (CNNs) perform poorly when applied to planning problems due to the reactive nature of the policies learned by them [5],[6],[7].”
BackgroundFast and flexible multilegged locomotion using learned centroidal dynamics
Kwon, T., Lee, Y., & van de Panne, M. (2020) · ACM Transactions on Graphics · doi:10.1145/3386569.3392432 · Cited by 51
“Trajectory optimization methods can be used to produce data for learning feedback-based controllers, typically as part of an iterative process, e.g., [Levine and Koltun 2013; Liu and Hodgins 2018; Liu et al. 2016, 2012; Mordatch et al. 2015; Zhang et al. 2016].”
MethodEstrada, P. M., Lima, D. D., Bauer, P. H., Mammetti, M., & Bruno, J. C. (2022) · Applied Energy · doi:10.1016/j.apenergy.2022.120231 · Cited by 50
Jiang, C., & Guo, Y. P. (2023) · IEEE Transactions on Control of Network Systems · doi:10.1109/tcns.2023.3290430 · Cited by 50
“An intuitive solution is to generate a reference trajectory that achieves good long-horizon performance and track the reference trajectory with receding horizon control, such as MPC [40].”
Data-driven nonlinear aeroelastic models of morphing wings for control
Fonzi, N., Brunton, S. L., & Fasel, U. (2020) · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rspa.2020.0079 · Cited by 50
“Nonlinear models based on machine learning, such as neural networks, are increasingly used due to advances in computing power, and recently deep reinforcement learning has been combined with MPC [74, 75], yielding impressive results in the large-data and high-performance computing limit.”
MethodAggressive Quadrotor Flight Using Curiosity-Driven Reinforcement Learning
Sun, Q., Fang, J., Zheng, W. X., & Tang, Y. S. (2022) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2022.3144586 · Cited by 49
“One of the promising methods to address this is reinforcement learning, through which UAVs can be navigated without the trajectory planning procedure [7] and which has been widely used in the control and navigation of agents [8]–[11].”
MethodCombining Reinforcement Learning with Model Predictive Control for On-Ramp Merging
Lubars, J., Gupta, H., Chinchali, S., Li, L., Raja, A., Srikant, R., & Wu, X. (2021) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc48978.2021.9564954 · Cited by 50
Holo-Dex: Teaching Dexterity with Immersive Mixed Reality
Arunachalam, S. P., Guzey, I., Chintala, S., & Pinto, L. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160547 · Cited by 75
“fect in robotics on a wide range of domains from manipulation [1, 2], locomotion [3, 4, 5], and aerial robotics [6, 7, 8].”
BackgroundLiu, M., Peeters, S., Callaway, D. S., & Claessens, B. J. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2018.2890275 · Cited by 48
Deep imitation learning for 3D navigation tasks
Hussein, A. M., Elyan, E., Gaber, M. M., & Jayne, C. (2017) · Neural Computing and Applications · doi:10.1007/s00521-017-3241-z · Cited by 57
“Since then several papers have proposed learning autonomous aerial navigation using demonstrations [22] and reinforcement learning [15, 16, 23].”
BackgroundPolvara, R., Sharma, S. K., Wan, J., Manning, A. J., & Sutton, R. (2019) · Robotica · doi:10.1017/s0263574719000316 · Cited by 47
Control of Quadrotors Using the Hopf Fibration on SO(3)
Watterson, M., & Kumar, V. (2019) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-28619-4_20 · Cited by 47
“There has been some research on learning based control [23], but the model based approaches are more mature and can provide provable guarantees on stability.”
BackgroundLearning Heuristic Search via ImitationPreprint
Bhardwaj, M., Choudhury, S., & Scherer, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.03034 · Cited by 47
“[36] extend guided policy search [37] for imitating MPC.”
Efficient Processing of Deep Neural Networks: A Tutorial and SurveyPreprint
Sze, V., Chen, Y., Yang, T.-J., & Emer, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.09039 · Cited by 45
“• Robotics DNNs have been successful in the domain of robotic tasks such as grasping with a robotic arm [32], motion planning for ground robots [33], visual navigation [4, 34], control to stabilize a quadcopter [35] and driving strategies for autonomous vehicles [36].”
Cyber-Physical Attack Threats Analysis for UAVs from CPS Perspective
Rong-xiao, G., Ji-wei, T., Bu-hong, W., & Fu-te, S. (2020) · International Conference on Computer Engineering and Application (ICCEA) · doi:10.1109/iccea50009.2020.00063 · Cited by 44
“The artificial intelligence algorithms such as deep learning have been widely used to the UAVs, especially in feature extraction [32][33], task recognition and path planning [34][35], autonomous flight control [36-40] and so on.”
MethodA Novel Reinforcement Learning-Based Robust Control Strategy for a Quadrotor
Hua, H., & Fang, Y. (2022) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2022.3165288 · Cited by 43
Metadata
- Authors
- Zhang, T., Kahn, G., Levine, S., & Abbeel, P.
- Affiliations
- University of California, Berkeley (US)
- Venue
- 2016 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2016-05
- Pages
- 528-535
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W1923344279
- Cited by
- 452
- References
- 45
- Citations by year
- 2026: 10 · 2025: 18 · 2024: 37 · 2023: 33 · 2022: 54 · 2021: 53 · 2020: 65 · 2019: 93 · 2018: 42 · 2017: 39 · 2016: 8
- FWCI
- 32.08
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (N66001)
- Army Research Office
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Physical Sciences › Engineering › Control and Systems Engineering › Advanced Control Systems Optimization
- Keywords
- model predictive control, deep neural network policies, autonomous aerial vehicles, quadrotor control, obstacle avoidance, reinforcement learning, state estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
