Target tracking control of UAV through deep reinforcement learning
Abstract
This study presents an innovative reinforcement-learning-based control algorithm for a vertical take-off and landing (VTOL) aircraft under wind disturbances. In our approach, the tracking control problem of the VTOL aircraft is formulated as a Markov decision process, and the appropriate system state, reward function, and soft update method are presented. To improve the control accuracy under wind disturbances, three kinds of wind fields were added in the learning environment to expand the exploration space and simulate the effect of wind disturbances on the flight control. Moreover, to ensure the tracking accuracy and the practical implementation, a quantum-inspired experience replay strategy was developed based on quantum computation theory. In this strategy, the preparation operation scheme was designed to encourage the exploration and speed up the convergence. The depreciation operation method was developed to enrich the sample diversity, which increased the robustness of the controller and allowed the control strategy learned in the numerical simulations to be directly transferred into real-world environments. Numerical simulations, hardware-in-the-loop experiments, and real-world flight experiments were conducted to evaluate the performance and merits of the proposed method. The results demonstrated high accuracy and effectiveness and good robustness of the proposed control algorithm in terms of standoff target tracking control and flight stability.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reinforcement learning eliminates the need for explicit mathematical models by autonomously learning optimal control policies through environmental interaction.
“As a representative methodology, reinforcement learning (RL) eliminates the need for explicit mathematical models by autonomously learning optimal control policies through environmental interaction [18,19].”
From Predefined-Time Tracking Control of Servo Hydraulic Cylinder Based on Reinforcement Learning (2025) · cites this paper as [18]
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control
Chu, T., Wang, J., Codeca, L., & Li, Z. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2901791 · Cited by 1,046 · see citing works
Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles
Aradi, S. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3024655 · Cited by 664 · see citing works
Fundamentals of Inertial Navigation, Satellite-based Positioning and their Integration
Noureldin, A. M., Karamat, T. B., & Georgy, J. (2012) · Springer eBooks · doi:10.1007/978-3-642-30466-8 · Cited by 632 · see citing works
Zhao, B., Xian, B., Zhang, Y., & Zhang, X. (2014) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2014.2364982 · Cited by 568 · see citing works
Reinforcement learning for control: Performance, stability, and deep approximators
Buşoniu, L., de Bruin, T., Tolić, D., Kober, J., & Palunko, I. (2018) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2018.09.005 · Cited by 468 · see citing works
Quantum Computers Can Search Arbitrarily Large Databases by a Single Query
Grover, L. K. (1997) · Physical Review Letters · doi:10.1103/physrevlett.79.4709 · Cited by 432 · see citing works
Quantum Reinforcement Learning
Dong, D., Chen, C., Li, H., & Tarn, T.-J. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.925743 · Cited by 410 · see citing works
Labbadi, M., & Cherkaoui, M. (2019) · ISA Transactions · doi:10.1016/j.isatra.2019.10.012 · Cited by 295 · see citing works
A survey on artificial intelligence trends in spacecraft guidance dynamics and control
Izzo, D., Märtens, M., & Pan, B. (2019) · Astrodynamics · doi:10.1007/s42064-018-0053-6 · Cited by 283 · see citing works
A novel DDPG method with prioritized experience replay
Hou, Y., Liu, L., Wei, Q., Xu, X., & Chen, C. (2017) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2017.8122622 · Cited by 277 · see citing works
Exploration and apprenticeship learning in reinforcement learning
Abbeel, P., & Ng, A. Y. (2005) · International Conference on Machine Learning · doi:10.1145/1102351.1102352 · Cited by 224 · see citing works
Active Disturbance Rejection Attitude Control for a Dual Closed-Loop Quadrotor Under Gust Wind
Yang, H., Cheng, L., Xia, Y., et al. (2017) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2017.2710951 · Cited by 220 · see citing works
Deep-Reinforcement-Learning-Based Autonomous UAV Navigation With Sparse Rewards
Wang, C., Wang, J., Wang, J., & Zhang, X. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2973193 · Cited by 219 · see citing works
Backstepping Sliding-Mode and Cascade Active Disturbance Rejection Control for a Quadrotor UAV
Xu, L., Ma, H., Guo, D., et al. (2020) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2020.2990582 · Cited by 216 · see citing works
Control of exploitation–exploration meta-parameter in reinforcement learning
Ishii, S., Yoshida, W., & Yoshimoto, J. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00056-4 · Cited by 209 · see citing works
Deterministic Policy Gradient With Integral Compensator for Robust Quadrotor Control
Wang, Y., Sun, J. L., He, H., & Sun, C. (2019) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2018.2884725 · Cited by 180 · see citing works
Al-Hilo, A., Samir, M., Assi, C. M., Sharafeddine, S., & Ebrahimi, D. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3020220 · Cited by 166 · see citing works
ICRA: An Intelligent Clustering Routing Approach for UAV Ad Hoc Networks
Guo, J., Gao, H., Liu, Z., Huang, F., Zhang, J., Li, X., & Ma, J. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3145857 · Cited by 165 · see citing works
Self-citationLabbadi, M., & Cherkaoui, M. (2021) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2021.3053109 · Cited by 159 · see citing works
Robust Backstepping Control Based on Integral Sliding Modes for Tracking of Quadrotors
Ramirez-Rodriguez, H., Parra‐Vega, V., Sanchez, A., et al. (2013) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-013-9909-4 · Cited by 142 · see citing works
Dong, H., Zhao, X., & Luo, B. (2020) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2020.3003797 · Cited by 122 · see citing works
Harmonious Lane Changing via Deep Reinforcement Learning
Wang, G., Hu, J., Li, Z., & Li, L. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3047129 · Cited by 119 · see citing works
Safe Exploration Algorithms for Reinforcement Learning Controllers
Mannucci, T., Kampen, E.-J. V., de Visser, C. C., & Chu, Q. P. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2654539 · Cited by 104 · see citing works
Mean Field Deep Reinforcement Learning for Fair and Efficient UAV Control
Chen, D., Qi, Q., Zhuang, Z., Wang, J., Liao, J., & Han, Z. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3008299 · Cited by 91 · see citing works
Muthusamy, P. K., Garratt, M., Pota, H. R., et al. (2021) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2021.3055170 · Cited by 91 · see citing works
Disturbance-Rejection-Based Optimized Robust Adaptive Controllers for UAVs
Kazim, M. Z., Azar, A. T., Koubâa, A., et al. (2021) · IEEE Systems Journal · doi:10.1109/jsyst.2020.3006059 · Cited by 86 · see citing works
Chen, L., Liu, Z., Dang, Q., Zhao, W., & Wang, G. (2022) · Aerospace Science and Technology · doi:10.1016/j.ast.2022.107749 · Cited by 79 · see citing works
Self-citationYamany, W., Moustafa, N., & Turnbull, B. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3130906 · Cited by 78 · see citing works
Dynamic Attitude Planning for Trajectory Tracking in Thrust-Vectoring UAVs
Invernizzi, D., Lovera, M., & Zaccarian, L. (2019) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2019.2919660 · Cited by 63 · see citing works
Vehicle Detection in Aerial Images Using Rotation-Invariant Cascaded Forest
Ma, B., Liu, Z., Jiang, F., et al. (2019) · IEEE Access · doi:10.1109/access.2019.2915368 · Cited by 60 · see citing works
Self-citationSafety Flight Control Design of a Quadrotor UAV With Capability Analysis
Zhou, X., Yu, X. Y., Guo, K., Zhou, S., Guo, L., Zhang, Y. M., & Peng, X. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3113168 · Cited by 58 · see citing works
Aggressive Quadrotor Flight Using Curiosity-Driven Reinforcement Learning
Sun, Q., Fang, J., Zheng, W. X., & Tang, Y. S. (2022) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2022.3144586 · Cited by 49 · see citing works
Lin, H., & Tang, C. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3121289 · Cited by 38 · see citing works
Exploring Potential Applications of Quantum Computing in Transportation Modelling
Cooper, C. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3132161 · Cited by 34 · see citing works
Quantum-enhanced feature selection with forward selection and backward elimination
He, Z., Li, L., Huang, Z., & Situ, H. (2018) · Quantum Information Processing · doi:10.1007/s11128-018-1924-8 · Cited by 32 · see citing works
Winkelmaier, G., Battulwar, R., Khoshdeli, M., Valencia, J., Sattarvand, J., & Parvin, B. (2020) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2020.2992011 · Cited by 32 · see citing works
Robust Trajectory Tracking for an Uncertain UAV Based on Active Disturbance Rejection
Blas, L. A., Davila, J. A., Salazar, S., & Bonilla, M. (2021) · IEEE Control Systems Letters · doi:10.1109/lcsys.2021.3111837 · Cited by 21 · see citing works
Li, Y., Li, H., Li, Z., Fang, H., Sanyal, A. K., Wang, Y., & Qiu, Q. (2019) · IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA) · doi:10.1109/rtcsa.2019.8864571 · Cited by 16 · see citing works
Using Soft Actor-Critic for Low-Level UAV ControlPreprint
Barros, G. M., & Colombini, E. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02293 · Cited by 14 · see citing works
Coordinated Turn of Fixed-Wing Aircraft Under Wind Interface on Integral Backstepping
Song, Y., Liu, Z., Han, J., et al. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2021.3122169 · Cited by 9 · see citing works
Self-citationEP-PQM: Efficient Parametric Probabilistic Quantum Memory With Fewer Qubits and Gates
Khan, M., Faye, J. P. L., Mendes, U. C., & Miranskyy, A. V. (2022) · IEEE Transactions on Quantum Engineering · doi:10.1109/tqe.2022.3169987 · Cited by 8 · see citing works
Zou, X., Liu, Z., Zhao, W. D., et al. (2021) · International Conference on Unmanned Aircraft Systems (ICUAS) · doi:10.1109/icuas51884.2021.9476862 · Cited by 7 · see citing works
Self-citationQuantum-enhanced reinforcement learning for control: a preliminary study
Hu, Y., Tang, F., Chen, J., & Wang, W. (2021) · Control Theory and Technology · doi:10.1007/s11768-021-00063-x · Cited by 5 · see citing works
Unmanned Aerial Vehicles Control Study Using Deep Deterministic Policy Gradient
Sun, D., Gao, D., Zheng, J., et al. (2018) · IEEE CSAA Guidance, Navigation and Control Conference (CGNCC) · doi:10.1109/gncc42960.2018.9018682 · Cited by 4 · see citing works
Research on Proximal Policy Optimization Algorithm Based on N-step Update
Guoqing, Z., Xu, J., Ai-dong, L., et al. (2021) · International Conference on Communications, Information System and Computer Engineering (CISCE) · doi:10.1109/cisce52179.2021.9445929 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Predefined-Time Tracking Control of Servo Hydraulic Cylinder Based on Reinforcement Learning
Han, T., Nie, X., Que, N., Lu, J., Yao, J., & Yu, X. (2025) · Actuators · doi:10.3390/act15010009 · Cited by 1 · see citing works
- Authors
- Ma, B., Liu, Z., Zhao, W., Yuan, J., Long, H., Wang, X., & Yuan, Z.
- Affiliations
- Northwestern Polytechnical University (CN)
- Beijing Union University (CN)
- Venue
- IEEE Transactions on Intelligent Transportation Systems
- Published
- 2023-06
- Volume
- 24
- Issue
- 6
- Pages
- 5983-6000
- ISSN
- 1524-9050 (ISSN-L)
- 1558-0016
- OpenAlex
- W4323338559
- Cited by
- 91
- References
- 47
- Citations by year
- 2026: 29 · 2025: 30 · 2024: 29 · 2023: 3
- FWCI
- 8.58
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Intelligent Transportation Systems (publishedVersion)
- Funders
- National Natural Science Foundation Fund (52072309)
- Key Research and Development Program of Shaanxi Province (2019ZDLGY14-02-01)
- Shenzhen Fundamental Research Program (JCYJ20190806152203506)
- Aeronautical Science Foundation of China (ASFC-2018ZC53026)
- Beijing Institute of Spacecraft System Engineering Research Project (JSZL2020203B004)
- Innovation Foundation for Doctor Dissertation of Northwestern Polytechnical University (CX2021033)
- Basic Research Program of Taicang (TC2021JC09)
- Natural Science Foundation of Shaanxi Province (2023-JC-QN-0003, 2023-JC-QN-0665)
- Premium Funding Project for Academic Human Resources Development in Beijing Union University (BPHR2020CZ03)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Aerospace Engineering › Aerospace Engineering and Control Systems
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Keywords
- UAV target tracking, vertical takeoff and landing aircraft, deep reinforcement learning, wind disturbance robustness, Markov decision process, reward function design, flight stability, hardware-in-the-loop testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
