The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Model-Free Optimal Tracking Control via Critic-Only Q-Learning
Abstract
Model-free control is an important and promising topic in control fields, which has attracted extensive attention in the past few years. In this paper, we aim to solve the model-free optimal tracking control problem of nonaffine nonlinear discrete-time systems. A critic-only Q-learning (CoQL) method is developed, which learns the optimal tracking control from real system data, and thus avoids solving the tracking Hamilton-Jacobi-Bellman equation. First, the Q-learning algorithm is proposed based on the augmented system, and its convergence is established. Using only one neural network for approximating the Q-function, the CoQL method is developed to implement the Q-learning algorithm. Furthermore, the convergence of the CoQL method is proved with the consideration of neural network approximation error. With the convergent Q-function obtained from the CoQL method, the adaptive optimal tracking control is designed based on the gradient descent scheme. Finally, the effectiveness of the developed CoQL method is demonstrated through simulation studies. The developed CoQL method learns with off-policy data and implements with a critic-only structure, thus it is easy to realize and overcome the inadequate exploration problem.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Neural Networks And Learning Machines
Haykin, S. (2010) · Cited by 7,535
Approximate Dynamic Programming: Solving the Curses of Dimensionality
Powell, W. B. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9781118029176 · Cited by 2,297
Hornik, K., Stinchcombe, M., & White, H. (1990) · Neural Networks · doi:10.1016/0893-6080(90)90005-6 · Cited by 2,111
Approximate Dynamic Programming: Solving the Curses of Dimensionality
Powell, W. B. (2007) · Wiley Series in Probability and Statistics · doi:10.1002/9780470182963 · Cited by 1,566
Reinforcement learning and adaptive dynamic programming for feedback control
Lewis, F. L., & Vrabie, D. (2009) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2009.933854 · Cited by 1,558
Neuro-Dynamic Programming: NDP
Bertsekas, D. P. (2001) · Encyclopedia of Optimization · doi:10.1007/0-306-48332-7_333 · Cited by 1,466
Jiang, Y., & Jiang, Z.-P. (2012) · Automatica · doi:10.1016/j.automatica.2012.06.096 · Cited by 1,072
Modares, H., & Lewis, F. L. (2014) · Automatica · doi:10.1016/j.automatica.2014.05.011 · Cited by 701
Zhang, H., Cui, L., Zhang, X., & Luo, Y. (2011) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2011.2168538 · Cited by 643
Al-Tamimi, A., Lewis, F. L., & Abu-Khalaf, M. (2007) · Automatica · doi:10.1016/j.automatica.2006.09.019 · Cited by 611
Kiumarsi, B., Lewis, F. L., Modares, H., Karimpour, A., & Naghibi-Sistani, M.-B. (2014) · Automatica · doi:10.1016/j.automatica.2014.02.015 · Cited by 574
Modares, H., Lewis, F. L., & Jiang, Z. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2441749 · Cited by 570
Lewis, F. L., & Vamvoudakis, K. G. (2011) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2010.2043839 · Cited by 544 · cited 1×
“Note that most of the existing related works [12], [24], [42], [51], [66], [68], [79], [80] belong to the on-policy learning framework or use on-policy implementation.”
BackgroundAdaptive attitude tracking control for rigid spacecraft with finite-time convergence
Lu, K., & Xia, Y. (2013) · Automatica · doi:10.1016/j.automatica.2013.09.001 · Cited by 512
Zhang, H., Wei, Q., & Luo, Y. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.920269 · Cited by 489
Asynchronous stochastic approximation and Q-learning
Tsitsiklis, J. N. (1994) · Machine Learning · doi:10.1007/bf00993306 · Cited by 453
Modares, H., & Lewis, F. L. (2014) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2014.2317301 · Cited by 449
Robust Adaptive Dynamic Programming and Feedback Stabilization of Nonlinear Systems
Jiang, Y., & Jiang, Z.-P. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2294968 · Cited by 414
Wang, D., Liu, D., Wei, Q., Zhao, D., & Jin, N. (2012) · Automatica · doi:10.1016/j.automatica.2012.05.049 · Cited by 394
Self-citationOff-Policy Reinforcement Learning for $ H_\infty $ Control Design
Luo, B., Wu, H.-N., & Huang, T. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2319577 · Cited by 362
Self-citationXu, B., Yang, C., & Pan, Y. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2456972 · Cited by 347
Actor–Critic-Based Optimal Tracking for Partially Unknown Nonlinear Discrete-Time Systems
Kiumarsi, B., & Lewis, F. L. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2358227 · Cited by 326
Even-Dar, E., & Mansour, Y. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_39 · Cited by 314
Incremental multi-step Q-learning
Peng, J., & Williams, R. J. (1996) · Machine Learning · doi:10.1007/bf00114731 · Cited by 286
Data-based approximate policy iteration for affine nonlinear continuous-time optimal control design
Luo, B., Wu, H.-N., Huang, T., & Liu, D. (2014) · Automatica · doi:10.1016/j.automatica.2014.10.056 · Cited by 280
Self-citationSpooner, J. T., Ordóñez, R., Maggiore, M., & Passino, K. M. (2001) · Medical Entomology and Zoology · Cited by 274
Approximate policy iteration: a survey and some new methods
Bertsekas, D. P. (2011) · Journal of Control Theory and Applications · doi:10.1007/s11768-011-1005-3 · Cited by 273
An overview on flight dynamics and control approaches for hypersonic vehicles
Xu, B., & Shi, Z. (2015) · Science China Information Sciences · doi:10.1007/s11432-014-5273-7 · Cited by 265
Reinforcement Learning Output Feedback NN Control Using Deterministic Learning Technique
Xu, B., Yang, C., & Shi, Z. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2292704 · Cited by 257
Optimal Tracking Control of Unknown Discrete-Time Linear Systems Using Input-Output Measured Data
Kiumarsi, B., Lewis, F. L., Naghibi-Sistani, M.-B., & Karimpour, A. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2384016 · Cited by 249
Finite-Horizon Control-Constrained Nonlinear Optimal Control Using Single Network Adaptive Critics
Heydari, A., & Balakrishnan, S. N. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2227339 · Cited by 246
Off-Policy Temporal Difference Learning with Function Approximation
Precup, D., Sutton, R. S., & Dasgupta, S. (2001) · Cited by 243
Approximate optimal trajectory tracking for continuous-time nonlinear systems
Kamalapurkar, R., Dinh, H., Bhasin, S., & Dixon, W. E. (2015) · Automatica · doi:10.1016/j.automatica.2014.10.103 · Cited by 232
Wei, Q., Liu, D., & Shi, G. (2014) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2014.2361485 · Cited by 230
Self-citationLiu, Y.-J., Tang, L., Tong, S., Chen, C. L. P., & Li, D.-J. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2360724 · Cited by 225
Wei, Q., & Liu, D. (2014) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2013.2284545 · Cited by 224
Self-citationPadhi, R., Unnikrishnan, N., Wang, X., & Balakrishnan, S. N. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.08.010 · Cited by 216
Wu, H., & Luo, B. (2012) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2217349 · Cited by 208
Self-citationZhang, H., Song, R., Wei, Q., & Zhang, T. (2011) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2011.2172628 · Cited by 191
Adaptive Learning in Tracking Control Based on the Dual Critic Network Design
Ni, Z., He, H., & Wen, J. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2247627 · Cited by 189
Finite-Approximation-Error-Based Discrete-Time Iterative Adaptive Dynamic Programming
Wei, Q., Wang, F.-Y., Liu, D., & Yang, X. (2014) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2354377 · Cited by 188
Self-citationLiu, D., Li, H., & Wang, D. (2013) · Neurocomputing · doi:10.1016/j.neucom.2012.11.021 · Cited by 188
Self-citationLee, J. Y., Park, J. B., & Choi, Y. H. (2012) · Automatica · doi:10.1016/j.automatica.2012.06.008 · Cited by 188
Wei, Q., & Liu, D. (2014) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2014.2301770 · Cited by 179
Self-citationToward Off-Policy Learning Control with Function Approximation
Maei, H. R., ri, C. S., Bhatnagar, S., & Sutton, R. (2010) · Cited by 175
Power System Stability Control for a Wind Farm Based on Adaptive Dynamic Programming
Tang, Y., He, H., Wen, J., & Liu, J. (2015) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2014.2346740 · Cited by 168
Even-DarEyal, & MansourYishay (2004) · Journal of Machine Learning Research · doi:10.5555/1005332.1005333 · Cited by 158
Zhong, X., He, H., Zhang, H., & Wang, Z. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2305841 · Cited by 153
Infinite Horizon Self-Learning Optimal Control of Nonaffine Discrete-Time Nonlinear Systems
Wei, Q., Liu, D., & Yang, X. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2401334 · Cited by 142
Self-citationLuo, B., Wu, H., & Li, H. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2320744 · Cited by 141
Self-citationNa, J., & Herrmann, G. (2014) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2014.7004668 · Cited by 140
A Novel Iterative $\theta $ -Adaptive Dynamic Programming for Discrete-Time Nonlinear Systems
Wei, Q., & Liu, D. (2013) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2013.2280974 · Cited by 138
Self-citationReinforcement learning solution for HJB equation arising in constrained optimal control problem
Luo, B., Wu, H., Huang, T., & Liu, D. (2015) · Neural Networks · doi:10.1016/j.neunet.2015.08.007 · Cited by 131
Self-citationLee, J. Y., Park, J. B., & Choi, Y. H. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2328590 · Cited by 127
Online Learning Control Using Adaptive Critic Designs With Sparse Kernel Machines
Xu, X., Hou, Z., Lian, C., & He, H. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2236354 · Cited by 126
Data-Driven $H_\infty$ Control for Nonlinear Distributed Parameter Systems
Luo, B., Huang, T., Wu, H., & Yang, X. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2461023 · Cited by 122
Self-citationQin, C., Zhang, H., & Luo, Y. (2013) · International Journal of Control · doi:10.1080/00207179.2013.863432 · Cited by 112
Model-Free Dual Heuristic Dynamic Programming
Ni, Z., He, H., Zhong, X., & Prokhorov, D. V. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2424971 · Cited by 108
Maei, H. R., & Sutton, R. S. (2010) · Artificial General Intelligence · doi:10.2991/agi.2010.22 · Cited by 106
Continuous-Time Q-Learning for Infinite-Horizon Discounted Cost Linear Quadratic Regulator Problems
Muthukumar, P., Modares, H., Lewis, F. L., & Aurangzeb, M. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2322116 · Cited by 97
Adaptive Learning and Control for MIMO System Based on Adaptive Dynamic Programming
Fu, J. Y., He, H., & Zhou, X. (2011) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2011.2147797 · Cited by 97
Model-free H ∞ control design for unknown linear discrete-time systems via Q-learning with LMI
Kim, J., & Lewis, F. L. (2010) · Automatica · doi:10.1016/j.automatica.2010.05.002 · Cited by 96
Kamalapurkar, R., Klotz, J. R., & Dixon, W. E. (2014) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2014.7004681 · Cited by 85
Luo, B., & Wu, H. (2012) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2012.2194781 · Cited by 82
Self-citationWei, Q., & Liu, D. (2014) · Neurocomputing · doi:10.1016/j.neucom.2013.09.069 · Cited by 79
Self-citationNumerical adaptive learning control scheme for discrete‐time non‐linear systems
Wei, Q., & Liu, D. (2013) · IET Control Theory and Applications · doi:10.1049/iet-cta.2012.0486 · Cited by 73
Self-citationOptimal Tracking Control of Motion Systems
Mannava, A., Balakrishnan, S. N., Tang, L. X., & Landers, R. G. (2011) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2011.2168608 · Cited by 55
Kiumarsi, B., Lewis, F. L., & Levine, D. S. (2015) · Neurocomputing · doi:10.1016/j.neucom.2014.12.067 · Cited by 47
Optimal tracking and two-channel disturbance rejection under control energy constraint
Wang, B., Guan, Z., & Yuan, F. (2011) · Automatica · doi:10.1016/j.automatica.2011.01.005 · Cited by 42
Luo, B., Wu, H., & Li, H. (2014) · Industrial & Engineering Chemistry Research · doi:10.1021/ie4031743 · Cited by 40
Self-citationNew algorithms of the Q-learning type
Bhatnagar, S., & Babu, K. M. (2007) · Automatica · doi:10.1016/j.automatica.2007.09.009 · Cited by 29
Cited by
Nguyen, T. T., Nguyen, N. D., & Nahavandi, S. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2977374 · Cited by 1,306
Adaptive Dynamic Programming for Control: A Survey and Recent Advances
Liu, D., Xue, S., Zhao, B., Luo, B., & Wei, Q. (2021) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2020.3042876 · Cited by 708
Self-citationWen, G., Chen, C. L. P., & Ge, S. S. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3002108 · Cited by 342
Adaptive Controller Design-Based ABLF for a Class of Nonlinear Time-Varying State Constraint Systems
Liu, Y.-J., Lu, S., Li, D., & Tong, S. (2017) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2016.2633007 · Cited by 284
Learning-Based Adaptive Optimal Tracking Control of Strict-Feedback Nonlinear Systems
Gao, W., & Jiang, Z.-P. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2761718 · Cited by 176
Metadata
- Authors
- Luo, B., Liu, D., Huang, T., & Wang, D.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2016-10
- Volume
- 27
- Issue
- 10
- Pages
- 2134-2144
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W2462102501
- Cited by
- 332
- References
- 76
- FWCI
- 31.26
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.