Reinforcement learning control for a class of discrete-time non-strict feedback multi-agent systems and application to multi-marine vehicles
Abstract
A novel control design problem for a class of non-strict feedback multi-agent systems (MAS) in discrete-time form is studied based on reinforcement learning (RL) and applied to multi-marine vehicles (MMV). Firstly, for this kind of discrete-time MAS, a novel system transformation, which can not only solve the noncausal problem that exists in the backstepping method but also reduce the computational complexity, is proposed. Secondly, the algebraic-loop problem inherent in the conventional controller design is solved by compensating the dynamics and using the property of neural network (NN). Thirdly, the multi-gradient recursive (MGR) RL scheme is developed for the sake of designing the optimal controller. Finally, the stability analysis is presented, and all signals are ensured to be semi-global uniformly ultimately bounded (SGUUB) in the Lyapunov's sense. Besides, this scheme is applied to the MMV which can be described in the non-strict feedback form to extend the application of the designed controller. The MMV simulation demonstrates the validation of this scheme.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reinforcement learning eliminates the need for explicit mathematical models by autonomously learning optimal control policies through environmental interaction.
“As a representative methodology, reinforcement learning (RL) eliminates the need for explicit mathematical models by autonomously learning optimal control policies through environmental interaction [18,19].”
From Predefined-Time Tracking Control of Servo Hydraulic Cylinder Based on Reinforcement Learning (2025) · cites this paper as [19]
Cites
An Overview of Recent Progress in the Study of Distributed Multi-Agent Coordination
Cao, Y., Yu, W., Ren, W., & Chen, G. (2012) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2012.2219061 · Cited by 2,399 · see citing works
On Consensus Algorithms for Double-Integrator Dynamics
Ren, W. (2008) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2008.924961 · Cited by 1,431 · see citing works
Event-based broadcasting for multi-agent average consensus
Seyboth, G. S., Dimarogonas, D. V., & Johansson, K. H. (2012) · Automatica · doi:10.1016/j.automatica.2012.08.042 · Cited by 1,247 · see citing works
Online learning control by association and reinforcement
Si, J., & Wang, Y.-T. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.914523 · Cited by 778 · see citing works
Necessary and Sufficient Conditions for Consensusability of Linear Multi-Agent Systems
Ma, C., & Zhang, J. (2010) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2010.2042764 · Cited by 763 · see citing works
Adaptive nonlinear control without overparametrization
Krstić, M., Kanellakopoulos, I., & Kokotović, P. V. (1992) · Systems & Control Letters · doi:10.1016/0167-6911(92)90111-5 · Cited by 728 · see citing works
Adaptive Dynamic Programming for Control: A Survey and Recent Advances
Liu, D., Xue, S., Zhao, B., Luo, B., & Wei, Q. (2020) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2020.3042876 · Cited by 714 · see citing works
Self-citationTong, S., Sun, K., & Sui, S. (2018) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2017.2686373 · Cited by 551 · see citing works
Adaptive Fuzzy Tracking Control Design for SISO Uncertain Nonstrict Feedback Nonlinear Systems
Tong, S., Li, Y., & Sui, S. (2016) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2016.2540058 · Cited by 491 · see citing works
Zhang, H., Jiang, H., Luo, Y., & Xiao, G. (2016) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2016.2542134 · Cited by 427 · see citing works
Liu, Y., Gao, Y., Tong, S., & Li, Y. (2015) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2015.2418000 · Cited by 409 · see citing works
Deception attacks on event-triggered distributed consensus estimation for nonlinear systems
Ren, H., Cheng, Z., Qin, J., & Lu, R. (2023) · Automatica · doi:10.1016/j.automatica.2023.111100 · Cited by 404 · see citing works
Peng, Z., Wang, J., & Wang, D. (2017) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2017.2699167 · Cited by 382 · see citing works
Finite-time consensus and collision avoidance control algorithms for multiple AUVs
Li, S., & Wang, X. (2013) · Automatica · doi:10.1016/j.automatica.2013.08.003 · Cited by 365 · see citing works
Shen, Q., Jiang, B., Shi, P. J., & Zhao, J. Y. (2013) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2013.2260757 · Cited by 347 · see citing works
Neural-Network-Based Adaptive Leader-Following Control for Multiagent Systems With Uncertainties
Cheng, L., Hou, Z., Tan, M., Lin, Y., & Zhang, W. (2010) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2010.2050601 · Cited by 344 · see citing works
Li, L., Wu, D., Huang, Y., & Yuan, Z. (2021) · Applied Ocean Research · doi:10.1016/j.apor.2021.102759 · Cited by 328 · see citing works
Adaptive Fuzzy Inverse Optimal Control for Uncertain Strict-Feedback Nonlinear Systems
Li, Y., Min, X., & Tong, S. (2019) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2019.2935693 · Cited by 317 · see citing works
Adaptive NN control for a class of strict-feedback discrete-time nonlinear systems
Ge, S. S., Li, G. Y., & Lee, T. H. (2003) · Automatica · doi:10.1016/s0005-1098(03)00032-3 · Cited by 269 · see citing works
NN Reinforcement Learning Adaptive Control for a Class of Nonstrict-Feedback Discrete-Time Systems
Bai, W., Li, T., & Tong, S. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2963849 · Cited by 254 · see citing works
Self-citationAdaptive Fixed-Time Control of Nonlinear MASs With Actuator Faults
Ren, H., Ma, H., Li, H., & Wang, Z. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123558 · Cited by 242 · see citing works
Yang, C., Ge, S. S., Xiang, C., Chai, T., & Lee, T. H. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2003290 · Cited by 237 · see citing works
Bai, W., Zhou, Q. F., Li, T., & Li, H. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2921057 · Cited by 233 · see citing works
Self-citationGe, S. S., Zhang, J., & Lee, T.-H. (2004) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2004.826827 · Cited by 228 · see citing works
Wen, G., Chen, C. L. P., Feng, J., & Zhou, N. (2017) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2017.2787561 · Cited by 218 · see citing works
Transportation 5.0: The DAO to Safe, Secure, and Sustainable Intelligent Transportation Systems
Wang, F., Lin, Y., Ioannou, P., Vlacic, L., Liu, X., Eskandarian, A., Lv, Y., Na, X., Cebon, D., Ma, J., et al. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3305380 · Cited by 216 · see citing works
Li, T., Bai, W., Liu, Q., Long, Y., & Chen, C. L. P. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3121403 · Cited by 211 · see citing works
Self-citationDistributed Optimal Consensus Control for Nonlinear Multiagent System With Unknown Dynamic
Zhang, J., Zhang, H., & Feng, T. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2728622 · Cited by 205 · see citing works
Distributed Consensus Control of One-Sided Lipschitz Nonlinear Multiagent Systems
Rehan, M., Jameel, A., & Ahn, C. K. (2017) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2017.2667701 · Cited by 157 · see citing works
Ren, C., Chen, L., & Chen, C. L. P. (2016) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2016.2554151 · Cited by 152 · see citing works
Reinforcement Learning-Based Output Feedback Control of Nonlinear Systems With Input Constraints
He, P., & Jagannathan, S. (2005) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2004.840124 · Cited by 124 · see citing works
Bai, W., Li, T., Long, Y., & Chen, C. L. P. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3094901 · Cited by 116 · see citing works
Self-citationModel-Free Dual Heuristic Dynamic Programming
Ni, Z., He, H., Zhong, X., & Prokhorov, D. V. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2424971 · Cited by 108 · see citing works
Yang, Q., Cao, W., Meng, W., & Si, J. (2021) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2021.3122802 · Cited by 108 · see citing works
Sun, J., & Liu, C. (2018) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2018.2859904 · Cited by 106 · see citing works
Consensus Algorithms for Double-integrator Dynamics
(2007) · Communications and control engineering/Communications and control engineering series · doi:10.1007/978-1-84800-015-5_4 · Cited by 104 · see citing works
Intermittent Control for Quasisynchronization of Delayed Discrete-Time Neural Networks
Ding, S., Wang, Z., & Rong, N. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3004894 · Cited by 98 · see citing works
Ning, J., Ma, Y., Li, T., Chen, C. L. P., & Tong, S. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3339852 · Cited by 82 · see citing works
Liu, D., Li, H., & Wang, D. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2402203 · Cited by 72 · see citing works
Bai, W., Zhang, W., Cao, L., & Liu, Q. (2023) · Ocean Engineering · doi:10.1016/j.oceaneng.2023.114545 · Cited by 70 · see citing works
Self-citationHu, X., Zhu, G., Ma, Y., Li, Z., Malekian, R., & Sotelo, M. Á. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3124635 · Cited by 69 · see citing works
GrHDP Solution for Optimal Consensus Control of Multiagent Discrete-Time Systems
Zhong, X., & He, H. (2018) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2018.2814018 · Cited by 60 · see citing works
Huang, Y., Wu, D., Li, L., & Feng, N. (2022) · Ocean Engineering · doi:10.1016/j.oceaneng.2022.110740 · Cited by 58 · see citing works
Bai, W., Li, T., Long, Y., Chen, C. L. P., Xiao, Y., Li, W., & Li, R. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3326466 · Cited by 34 · see citing works
Self-citationShih, P., Kaul, B. C., Jagannathan, S., & Drallmeier, J. A. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2000452 · Cited by 32 · see citing works
Bai, W., Ren, J., & Li, T. (2018) · China Ocean Engineering · doi:10.1007/s13344-018-0030-0 · Cited by 31 · see citing works
Self-citationWei, T., Zhu, M., Chen, S., Cheng, X., Wen, Y., Zhang, W., Negenborn, R. R., & Pang, Y. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3220000 · Cited by 29 · see citing works
A Reinforcement Learning Scheme for a Partially-Observable Multi-Agent Game
Ishii, S., Fujita, H., Mitsutake, M., Yamazaki, T., Matsuda, J., & Matsuno, Y. (2005) · Machine Learning · doi:10.1007/s10994-005-0461-8 · Cited by 24 · see citing works
Introduction to discrete-time reinforcement learning control in Complex Engineering Systems
Bai, W. (2024) · Complex Engineering Systems · doi:10.20517/ces.2024.18 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Yao, D., Zheng, Z., Ren, H., Li, H., & Shi, Y. (2025) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2025.3586510 · Cited by 32 · see citing works
Wang, Y., An, T., Dong, B., Zhu, M., & Li, Y. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3527569 · Cited by 14 · see citing works
ADP-based adaptive control of ship course tracking system with prescribed performance
Li, P., Bai, W., Zhang, W., & Zhao, Y. (2024) · Proceedings of the Institution of Mechanical Engineers Part I Journal of Systems and Control Engineering · doi:10.1177/09596518241286422 · Cited by 9 · see citing works
Self-citationModel selection for predicting marine traffic flow in coastal waterways using deep learning methods
Wang, J., Wang, X., & Zhou, J. (2025) · Ocean Engineering · doi:10.1016/j.oceaneng.2025.121151 · Cited by 9 · see citing works
Optimal Tracking Control of Uncertain Nonlinear Systems Using Simplified Reinforcement Learning
Ning, P., Duan, L., & Hua, C. (2026) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2026.3650813 · Cited by 7 · see citing works
Zheng, J., Huang, Z., Wang, X., Qi, X., & Chen, X. (2025) · Ocean Engineering · doi:10.1016/j.oceaneng.2025.122559 · Cited by 6 · see citing works
Zheng, J., Guo, Y., Rong, W., Zhou, W., Wang, X., & Qi, X. (2025) · Ocean Engineering · doi:10.1016/j.oceaneng.2025.122913 · Cited by 5 · see citing works
Hong, Y., Xue, S., Liu, D., & Wang, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129673 · Cited by 5 · see citing works
Reinforcement Learning-Based Optimal Control for USVs Course Tracking under Disturbances
Li, P., Bai, W., Zhang, W., & Xv, Y. (2025) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-025-02281-w · Cited by 5 · see citing works
Self-citationOptimizing Informer with Whale Optimization Algorithm for Enhanced Ship Trajectory Prediction
Xie, H., Wang, J., Shi, Z., & Xue, S. (2025) · Journal of Marine Science and Engineering · doi:10.3390/jmse13101999 · Cited by 4 · see citing works
Resource state adaptive collaboration mechanism based on resource modeling and multi-agent system
Li, Z., Piao, C., Chu, D., Tu, Z., Hu, X., & Ding, D. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-01882-0 · Cited by 4 · see citing works
Özcan, A. B., Bayezit, İ., Kınacı, Ö. K., & Khan, J. A. (2026) · Ocean Engineering · doi:10.1016/j.oceaneng.2026.124201 · Cited by 3 · see citing works
Identifier-Critic-Based Adaptive Optimal Control for USVs With Guaranteed Performance
Chen, L., Wang, Y., Miao, Z., Jiang, Y., Wang, X., & Dai, S.-L. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3640155 · Cited by 2 · see citing works
Liu, S., Qiao, B., Liu, Z., Zhao, Z., & He, W. (2026) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2026.3662473 · Cited by 2 · see citing works
Predefined-Time Tracking Control of Servo Hydraulic Cylinder Based on Reinforcement Learning
Han, T., Nie, X., Que, N., Lu, J., Yao, J., & Yu, X. (2025) · Actuators · doi:10.3390/act15010009 · Cited by 1 · see citing works
Shi, J., Li, Z., Wei, Y., Huang, X., & Gao, S. (2025) · Proceedings of the Institution of Mechanical Engineers Part I Journal of Systems and Control Engineering · doi:10.1177/09596518251346046 · Cited by 1 · see citing works
Stability of Time-Varying Switched Homogeneous Systems with Degree Less Than One
Liu, B. Y., & Tian, Y. (2025) · Circuits Systems and Signal Processing · doi:10.1007/s00034-025-03246-y · Cited by 1 · see citing works
Synchronization in Multi-Layer Networks With Aperiodically Intermittent Adaptive Coupling
Ding, S., Hu, L., & Rong, N. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3645419 · Cited by 1 · see citing works
., A., & Malik, A. (2026) · Peer-to-Peer Networking and Applications · doi:10.1007/s12083-025-02170-x · Cited by 1 · see citing works
Liang, J., An, T., Ma, B., Shan, Z., & Dong, B. (2026) · Applied Mathematical Modelling · doi:10.1016/j.apm.2026.116911 · Cited by 1 · see citing works
Cooperative Optimization Hunting Control of Multi‐MSV Based on Reinforcement Learning
Wang, Y., Bai, W., & Zhang, W. (2025) · Applied Computational Intelligence and Soft Computing · doi:10.1155/acis/1415549
Self-citationReinforcement learning control for systems with unknown coupling induced by the compensator
Cheng, S., Li, Y., Wang, K., & Yang, C. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103449
Zhang, K., Zhang, W., & Du, X. (2025) · Ships and Offshore Structures · doi:10.1080/17445302.2025.2552881
He, W., Wang, Y., & Shi, Y. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108602
Ma, Y., Zhang, K., & Jiang, B. (2026) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2026.3704612
Liu, Z., Yao, Y., Kang, Y., Zhao, Y., Tan, J., Gu, L., & Li, Q. (2026) · Communications in Nonlinear Science and Numerical Simulation · doi:10.1016/j.cnsns.2026.110407
Wang, L., He, W., & Qian, F. (2026) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2026.3701960
Intermittent adaptive coupling for synchronization of complex networks with unknown node dynamics
Hu, L., Ding, S., & Rong, N. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134354
Deng, Y., Liu, F., Wang, J., Yan, J., & Xu, Y. (2026) · Journal of Ocean Engineering and Science · doi:10.1016/j.joes.2026.07.006
Reinforcement Learning Control for Unknown Nonlinear Systems Based on Adaptive Deep Neural Networks
Wang, Y., Li, S., Bai, W., & Lei, J. (2026) · Youth Academic Annual Conference of Chinese Association of Automation (YAC) · doi:10.1109/yac71005.2026.11616172
Self-citationDwivedi, A., Karan, A., Ali, A., & Singh, L. K. (2026) · International Journal of Robust and Nonlinear Control · doi:10.1002/rnc.70720
- Authors
- Bai, W., Chen, D., Zhao, B., & D'Ariano, A.
- Affiliations
- Dalian Maritime University (CN)
- Fujian University of Technology (CN)
- Beijing Normal University (CN)
- Department of Civil, Computer Science and Aeronautical Technologies Engineering, Roma Tre University, Rome, Italy
- Venue
- IEEE Transactions on Intelligent Vehicles
- Published
- 2025-05
- Volume
- 10
- Issue
- 5
- Pages
- 3613-3625
- ISSN
- 2379-8858 (ISSN-L)
- 2379-8904
- OpenAlex
- W4402435285
- Cited by
- 31
- References
- 49
- Citations by year
- 2026: 13 · 2025: 17 · 2024: 1
- FWCI
- 6.50
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Intelligent Vehicles (publishedVersion)
- Iris (Roma Tre University) (submittedVersion)
- Funders
- National Natural Science Foundation of China (52271360, 61976055, 61973330, 61903092)
- Dalian Outstanding Young Scientific and Technological Talents Project (2023RY031)
- Third Batch of Innovative Star Talent Plan in Fujian Province (003002)
- Basic Scientific Research Project of Liaoning Education Department (JYTMS20230164)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Control of Nonlinear Systems
- Keywords
- reinforcement learning control, multi-agent systems, nonstrict-feedback systems, discrete-time systems, backstepping control, neural network compensation, semi-global uniform ultimate boundedness, stability analysis, systemic change
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
