Reinforcement learning-based optimal tracking control of an unknown unmanned surface vehicle
Abstract
In this article, a novel reinforcement learning-based optimal tracking control (RLOTC) scheme is established for an unmanned surface vehicle (USV) in the presence of complex unknowns, including dead-zone input nonlinearities, system dynamics, and disturbances. To be specific, dead-zone nonlinearities are decoupled to be input-dependent sloped controls and unknown biases that are encapsulated into lumped unknowns within tracking error dynamics. Neural network (NN) approximators are further deployed to adaptively identify complex unknowns and facilitate a Hamilton-Jacobi-Bellman (HJB) equation that formulates optimal tracking. In order to derive a practically optimal solution, an actor-critic reinforcement learning framework is built by employing adaptive NN identifiers to recursively approximate the total optimal policy and cost function. Eventually, theoretical analysis shows that the entire RLOTC scheme can render tracking errors that converge to an arbitrarily small neighborhood of the origin, subject to optimal cost. Simulation results and comprehensive comparisons on a prototype USV demonstrate remarkable effectiveness and superiority.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences
Werbos, P. J. (1974) · Medical Entomology and Zoology · Cited by 2,772 · see citing works
Beyond regression : new fools for prediction and analysis in the behavioral sciences
Werbos, P. J. (1974) · Medical Entomology and Zoology · Cited by 2,746 · see citing works
Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem
Vamvoudakis, K. G., & Lewis, F. L. (2010) · Automatica · doi:10.1016/j.automatica.2010.02.018 · Cited by 1,722 · see citing works
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Medical Entomology and Zoology · Cited by 971 · see citing works
Adaptive maneuvering, with experiments, for a model ship in a marine control laboratory
Skjetne, R., Fossen, T. I., & Kokotović, P. V. (2004) · Automatica · doi:10.1016/j.automatica.2004.10.006 · Cited by 877 · see citing works
Bhasin, S., Kamalapurkar, R., Johnson, M. L., Vamvoudakis, K. G., Lewis, F. L., & Dixon, W. E. (2012) · Automatica · doi:10.1016/j.automatica.2012.09.019 · Cited by 646 · see citing works
Direct adaptive NN control of a class of nonlinear systems
Ge, S. S., & Wang, C. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/72.977306 · Cited by 621 · see citing works
Cooperative control of a nonuniform gantry crane with constrained tension
He, W., & Ge, S. S. (2016) · Automatica · doi:10.1016/j.automatica.2015.12.026 · Cited by 550 · see citing works
Vibration Control of a Flexible Robotic Manipulator in the Presence of Input Deadzone
He, W., Ouyang, Y., & Hong, J. (2016) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2016.2608739 · Cited by 394 · see citing works
Observer-Based Adaptive Fuzzy Fault-Tolerant Optimal Control for SISO Nonlinear Systems
Li, Y., Sun, K., & Tong, S. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2785801 · Cited by 379 · see citing works
Adaptive Robust Finite-Time Trajectory Tracking Control of Fully Actuated Marine Surface Vehicles
Wang, N., Qian, C., Sun, J., & Liu, Y. (2015) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2015.2496585 · Cited by 346 · see citing works
Self-citationIterative Learning Control for a Flapping Wing Micro Aerial Vehicle Under Distributed Disturbances
He, W., Meng, T., He, X., et al. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2808321 · Cited by 331 · see citing works
Platoon Formation Control With Prescribed Performance Guarantees for USVs
Dai, S., He, S., Lin, H., et al. (2017) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2017.2758743 · Cited by 328 · see citing works
Adaptive Neural Control of Underactuated Surface Vessels With Prescribed Performance Guarantees
Dai, S., He, S., Wang, M., et al. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2876685 · Cited by 324 · see citing works
Neural Learning Control of Marine Surface Vessels With Guaranteed Transient Tracking Performance
Dai, S., Wang, M., & Wang, C. (2015) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2015.2504553 · Cited by 315 · see citing works
Li, Y., Tong, S., & Li, T. (2013) · Fuzzy Sets and Systems · doi:10.1016/j.fss.2013.11.006 · Cited by 270 · see citing works
Adaptive Tracking Control of Surface Vessel Using Optimized Backstepping Technique
Wen, G., Ge, S. S., Chen, C. L. P., Tu, F., & Wang, S. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2844177 · Cited by 269 · see citing works
An Event-Triggered ADP Control Approach for Continuous-Time System With Unknown Internal States
Zhong, X., & He, H. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2523878 · Cited by 263 · see citing works
Deep Reinforcement Learning-Based Automatic Exploration for Navigation in Unknown Environment
Li, H., Zhang, Q., & Zhao, D. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2927869 · Cited by 262 · see citing works
Accurate Trajectory Tracking of Disturbed Surface Vehicles: A Finite-Time Control Approach
Wang, N., Karimi, H. R., Li, H., & Su, S. (2019) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2019.2906395 · Cited by 258 · see citing works
Self-citationBoundary Disturbance Observer-Based Control of a Vibrating Single-Link Flexible Manipulator
Zhao, Z., He, X., & Ahn, C. K. (2019) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2019.2912900 · Cited by 257 · see citing works
Self-citationValue and Policy Iterations in Optimal Control and Adaptive Dynamic Programming
Bertsekas, D. P. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2503980 · Cited by 239 · see citing works
Wang, D., Liu, D., & Li, H. (2014) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2013.2296206 · Cited by 238 · see citing works
Xu, B. (2015) · Nonlinear Dynamics · doi:10.1007/s11071-015-1958-8 · Cited by 235 · see citing works
Zhu, Y., Zhao, D., He, H., et al. (2016) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2016.2597763 · Cited by 229 · see citing works
Wang, Y., Jiang, B., Wu, Z., et al. (2020) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2020.2964808 · Cited by 225 · see citing works
Dynamic Learning From Adaptive Neural Network Control of a Class of Nonaffine Nonlinear Systems
Dai, S., Wang, C., & Wang, M. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2257843 · Cited by 222 · see citing works
Adaptive Fuzzy Control of Nonlinear Systems With Unknown Dead Zones Based on Command Filtering
Yu, J., Shi, P. J., Dong, W., & Lin, C. (2016) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2016.2634162 · Cited by 222 · see citing works
He, H., Ni, Z., & Fu, J. Y. (2011) · Neurocomputing · doi:10.1016/j.neucom.2011.05.031 · Cited by 221 · see citing works
Data-Based Adaptive Critic Designs for Nonlinear Robust Optimal Control With Uncertain Dynamics
Wang, D., Liu, D., Zhang, Q., et al. (2015) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2015.2492941 · Cited by 217 · see citing works
Boundary Adaptive Robust Control of a Flexible Riser System With Input Nonlinearities
Zhao, Z., He, X., Ren, Z., et al. (2018) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2018.2882734 · Cited by 214 · see citing works
Event-Based Robust Control for Uncertain Nonlinear Systems Using Adaptive Dynamic Programming
Zhang, Q., Zhao, D., & Wang, D. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2614002 · Cited by 204 · see citing works
Wang, N., & Er, M. J. (2016) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2015.2510587 · Cited by 201 · see citing works
Self-citationAdaptive Robust Online Constructive Fuzzy Control of a Complex Surface Vehicle System
Wang, N., Er, M. J., Sun, J., & Liu, Y. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2451116 · Cited by 200 · see citing works
Self-citationDeep Reinforcement Learning With Visual Attention for Vehicle Classification
Zhao, D., Chen, Y., & Lv, L. (2016) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2016.2614675 · Cited by 197 · see citing works
Wang, Y., Yang, X., & Yan, H. (2019) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2019.2892696 · Cited by 194 · see citing works
Wang, N., & Er, M. J. (2014) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2014.2359880 · Cited by 170 · see citing works
Self-citationWang, N., Su, S., Yin, J., et al. (2017) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2017.2737405 · Cited by 168 · see citing works
Self-citationDynamics-Constrained Global-Local Hybrid Path Planning of an Autonomous Surface Vehicle
Wang, N., & Xu, H. (2020) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2020.2991220 · Cited by 163 · see citing works
Self-citationWang, N., & Su, S. (2019) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2019.2955657 · Cited by 148 · see citing works
Self-citationBoundary Antidisturbance Control of a Spatially Nonlinear Flexible String System
Zhao, Z., Ahn, C. K., & Li, H. (2019) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2019.2931230 · Cited by 146 · see citing works
Self-citationQin, H., Chen, H., Sun, Y., et al. (2019) · Ocean Engineering · doi:10.1016/j.oceaneng.2019.106341 · Cited by 143 · see citing works
Wang, N., & Deng, Z. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2930471 · Cited by 139 · see citing works
Self-citationWang, N., Sun, J., & Er, M. J. (2017) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2017.2697399 · Cited by 134 · see citing works
Self-citationReliable Intelligent Path Following Control for a Robotic Airship Against Sensor Faults
Wang, Y., Zhou, W., Luo, J., Yan, H., Pu, H., & Peng, Y. (2019) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2019.2929224 · Cited by 134 · see citing works
Yaw-Guided Trajectory Tracking Control of an Asymmetric Underactuated Surface Vehicle
Wang, N., Su, S., Pan, X., et al. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2877046 · Cited by 127 · see citing works
Self-citationData-Based Reinforcement Learning for Nonzero-Sum Games With Unknown Drift Dynamics
Zhang, Q., & Zhao, D. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2830820 · Cited by 119 · see citing works
Successive Waypoints Tracking of an Underactuated Surface Vehicle
Wang, N., & Karimi, H. R. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2922823 · Cited by 115 · see citing works
Self-citationHyperbolic-Tangent LOS Guidance-Based Finite-Time Path Following of Underactuated Marine Vehicles
Wang, N., & Ahn, C. K. (2019) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2019.2947845 · Cited by 115 · see citing works
Self-citationAdaptive Learning and Control for MIMO System Based on Adaptive Dynamic Programming
Fu, J. Y., He, H., & Zhou, X. (2011) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2011.2147797 · Cited by 96 · see citing works
Full-State Regulation Control of Asymmetric Underactuated Surface Vehicles
Wang, N., Xie, G., Pan, X., et al. (2019) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2018.2890500 · Cited by 90 · see citing works
Self-citationGlobal asymptotic output tracking of nonlinear second-order systems with power integrators
Wang, N., Qian, C., & Sun, Z. (2017) · Automatica · doi:10.1016/j.automatica.2017.02.026 · Cited by 84 · see citing works
Self-citationPath Following of Autonomous Underactuated Ships: A Translation–Rotation Cascade Control Approach
Wang, N., & Pan, X. (2019) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2019.2932205 · Cited by 84 · see citing works
Self-citationAdaptive Near-Optimal Control of Uncertain Systems With Application to Underactuated Surface Vessels
Zhang, Y., Li, S., & Liu, X. (2017) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2017.2705057 · Cited by 81 · see citing works
Dynamics-Level Finite-Time Fuzzy Monocular Visual Servo of an Unmanned Surface Vehicle
Wang, N., & He, H. (2019) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2019.2952786 · Cited by 80 · see citing works
Self-citationTutsoy, Ö., Barkana, D. E., & Tugal, H. (2018) · ISA Transactions · doi:10.1016/j.isatra.2018.03.002 · Cited by 73 · see citing works
Adaptive backstepping control for air-breathing hypersonic vehicles with input nonlinearities
Hu, Q., Meng, Y., Wang, C., et al. (2017) · Aerospace Science and Technology · doi:10.1016/j.ast.2017.12.001 · Cited by 70 · see citing works
Stability Analysis of Optimal Adaptive Control Using Value Iteration With Approximation Errors
Heydari, A. (2018) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2018.2790260 · Cited by 67 · see citing works
Weng, Y., Wang, N., & Soares, C. G. (2020) · Ocean Engineering · doi:10.1016/j.oceaneng.2019.106910 · Cited by 54 · see citing works
Self-citationVamvoudakis, K. G., Vrabie, D., & Lewis, F. L. (2009) · IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning · doi:10.1109/adprl.2009.4927523 · Cited by 46 · see citing works
Tutsoy, Ö. (2015) · Asian Journal of Control · doi:10.1002/asjc.1184 · Cited by 46 · see citing works
Fuzzy-Based Goal Representation Adaptive Dynamic Programming
Tang, Y., He, H., Ni, Z., et al. (2015) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2015.2505327 · Cited by 46 · see citing works
Wang, N., Gao, Y., Sun, Z. X., et al. (2017) · International Journal of Fuzzy Systems · doi:10.1007/s40815-017-0387-x · Cited by 44 · see citing works
Self-citationSu, X., Liu, Z., & Lai, G. (2018) · Nonlinear Dynamics · doi:10.1007/s11071-017-3984-1 · Cited by 43 · see citing works
An expectation-maximization based single-beacon underwater navigation method with unknown ESV
Qin, H., Yu, X., Zhu, Z., et al. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.10.066 · Cited by 39 · see citing works
Zhao, J., Gan, M., & Zhang, C. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.06.090 · Cited by 30 · see citing works
Gao, Y., Tong, S., & Li, Y. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.04.078 · Cited by 22 · see citing works
Self-citationGan, M., Zhao, J., & Zhang, C. (2019) · Asian Journal of Control · doi:10.1002/asjc.2243 · Cited by 20 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Chen, R.-Z., Li, Y.-X., & Ahn, C. K. (2023) · Aerospace Science and Technology · doi:10.1016/j.ast.2022.108060 · Cited by 43 · see citing works
Self-citationOptimized-Based Fault-Tolerant Control of an Electro-Hydraulic System with Disturbance Rejection
Phan, V. D., & Ahn, K. K. (2022) · Applied Sciences · doi:10.3390/app12189197 · Cited by 17 · see citing works
Predefined-Time Tracking Control of Servo Hydraulic Cylinder Based on Reinforcement Learning
Han, T., Nie, X., Que, N., Lu, J., Yao, J., & Yu, X. (2025) · Actuators · doi:10.3390/act15010009 · Cited by 1 · see citing works
- Authors
- Wang, N., Gao, Y., Zhao, H., & Ahn, C. K.
- Affiliations
- Dalian Maritime University (CN)
- Korea University (KR)
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2021-07
- Volume
- 32
- Issue
- 7
- Pages
- 3034-3045
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W3047221759
- Cited by
- 322
- References
- 68
- Citations by year
- 2026: 35 · 2025: 74 · 2024: 81 · 2023: 65 · 2022: 40 · 2021: 23 · 2020: 4
- FWCI
- 16.39
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Natural Science Foundation of China (51009017, 51379002)
- Fund for Liaoning Innovative Talents in Colleges and Universities, China (LR2017024)
- Fund for Dalian Distinguished Young Scholars, China (2016RJ10)
- Liaoning Revitalization Talents Program (XLYC1807013)
- Stable Supporting Fund of Science and Technology on Underwater Vehicle Laboratory, China (SXJQR2018WDKT03)
- Fundamental Research Funds for the Central Universities, China (3132019344)
- National Research Foundation of Korea
- Korean Government (NRF-2020R1A2C1005449)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Control of Nonlinear Systems
- Keywords
- unmanned surface vehicles, optimal tracking control, reinforcement learning, dead-zone nonlinearity, unknown dynamics, disturbance compensation, Hamilton-Jacobi-Bellman equation, actor-critic, neural network identifier, adaptive identification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
