The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Off-Policy Reinforcement Learning for $ H_\infty $ Control Design
Open accessAbstract
The H∞ control design problem is considered for nonlinear systems with unknown internal system model. It is known that the nonlinear H∞ control problem can be transformed into solving the so-called Hamilton-Jacobi-Isaacs (HJI) equation, which is a nonlinear partial differential equation that is generally impossible to be solved analytically. Even worse, model-based approaches cannot be used for approximately solving HJI equation, when the accurate system model is unavailable or costly to obtain in practice. To overcome these difficulties, an off-policy reinforcement leaning (RL) method is introduced to learn the solution of HJI equation from real system data instead of mathematical system model, and its convergence is proved. In the off-policy RL method, the system data can be generated with arbitrary policies rather than the evaluating policy, which is extremely important and promising for practical systems. For implementation purpose, a neural network (NN)-based actor-critic structure is employed and a least-square NN weight update algorithm is derived based on the method of weighted residuals. Finally, the developed NN-based off-policy RL method is tested on a linear F16 aircraft plant, and further applied to a rotational/translational actuator system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Slotine, J.-J. E., & Li, W. (1991) · CERN Document Server (European Organization for Nuclear Research) · Cited by 18,975
(1995) · Cited by 10,558
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Weiss, M. (1997) · Automatica · doi:10.1016/s0005-1098(97)00132-5 · Cited by 5,495
Zhou, K., Doyle, J. C., & Glover, K. (1997) · Cited by 4,556
Doyle, J. C. (2002) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.1996.572756 · Cited by 3,742
Aircraft Control and Simulation
Stevens, B., & Lewis, F. L. (2004) · Aircraft Engineering and Aerospace Technology · doi:10.1108/aeat.2004.12776eae.001 · Cited by 3,167
Approximate Dynamic Programming: Solving the Curses of Dimensionality
Powell, W. B. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9781118029176 · Cited by 2,297
The Method of Weighted Residuals and Variational Principles
Finlayson, B. A. (2013) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611973242 · Cited by 1,945
Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem
Vamvoudakis, K. G., & Lewis, F. L. (2010) · Automatica · doi:10.1016/j.automatica.2010.02.018 · Cited by 1,703
A new algorithm for adaptive multidimensional integration
Peter Lepage, G. (1978) · Journal of Computational Physics · doi:10.1016/0021-9991(78)90004-9 · Cited by 1,657
L/sub 2/-gain analysis of nonlinear systems and nonlinear state-feedback H/sub infinity / control
van der Schaft, A. J. (1992) · IEEE Transactions on Automatic Control · doi:10.1109/9.256331 · Cited by 1,587
Approximate Dynamic Programming: Solving the Curses of Dimensionality
Powell, W. B. (2007) · Wiley Series in Probability and Statistics · doi:10.1002/9780470182963 · Cited by 1,566
Abu-Khalaf, M., & Lewis, F. L. (2005) · Automatica · doi:10.1016/j.automatica.2004.11.034 · Cited by 1,349
(1995) · Control Engineering Practice · doi:10.1016/0967-0661(95)90063-2 · Cited by 1,310
On an iterative technique for Riccati equation computations
Kleinman, D. (1968) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1968.1098829 · Cited by 1,210
H/sup ∞/-0ptimal Control and Related Minimax Design Problems: A Dynamic Game Approach
Basar, T., & Bernhard, P. (1996) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1996.536519 · Cited by 1,047
Adaptive optimal control for continuous-time linear systems based on policy iteration
Vrabie, D., Pastravanu, O., Abu-Khalaf, M., & Lewis, F. L. (2009) · Automatica · doi:10.1016/j.automatica.2008.08.017 · Cited by 949
Disturbance attenuation and H/sub infinity /-control via measurement feedback in nonlinear systems
Isidori, A., & Astolfi, A. (1992) · IEEE Transactions on Automatic Control · doi:10.1109/9.159566 · Cited by 885
Policy Iteration Adaptive Dynamic Programming Algorithm for Discrete-Time Nonlinear Systems
Liu, D., & Wei, Q. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2281663 · Cited by 753
Murray, J. J., Cox, C. J., Lendaris, G. G., & Saeks, R. (2002) · IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2002.801727 · Cited by 677 · cited 3×
“[5] presented two PI algorithms that avoid the necessity of knowing the internal system dynamics.”
BackgroundGalerkin approximations of the generalized Hamilton-Jacobi-Bellman equation
Beard, R. W., Saridis, G. N., & Wen, J. T.-Y. (1997) · Automatica · doi:10.1016/s0005-1098(97)00128-3 · Cited by 667
Approximate dynamic programming: solving the curses of dimensionality
Pardalos, P. M. (2009) · Optimization Methods and Software · doi:10.1080/10556780802583108 · Cited by 666
Zhang, H., Luo, Y., & Liu, D. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2009.2027233 · Cited by 649
Zhang, H., Cui, L., Zhang, X., & Luo, Y. (2011) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2011.2168538 · Cited by 643
Vrabie, D., & Lewis, F. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.03.008 · Cited by 634
Farrell, J. A., & Polycarpou, M. M. (2006) · Wiley eBooks · doi:10.1002/0471781819 · Cited by 617
Modares, H., Lewis, F. L., & Naghibi-Sistani, M.-B. (2014) · Automatica · doi:10.1016/j.automatica.2013.09.043 · Cited by 570
L2-Gain and Passivity Techniques in Nonlinear Control
van der Schaft, A. J. (1996) · Lecture notes in control and information sciences · doi:10.1007/3-540-76074-1 · Cited by 531
Vamvoudakis, K. G., & Lewis, F. L. (2011) · Automatica · doi:10.1016/j.automatica.2011.03.005 · Cited by 525 · cited 1×
“In fact, this problem also widely arises in many existing works for solving optimal and H∞ control problems of either linear or nonlinear systems through the observations from computer simulation, such as [5], [8], [9], [27], [30], [31], [36], [40], [42], and [56]–[59].”
BackgroundZhang, H., Wei, Q., & Luo, Y. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.920269 · Cited by 489
Zhang, H., Cui, L., & Luo, Y. (2013) · IEEE Transactions on Cybernetics · doi:10.1109/tsmcb.2012.2203336 · Cited by 464
Zhang, H., Wei, Q., & Liu, D. (2011) · Automatica · doi:10.1016/j.automatica.2010.10.033 · Cited by 409
Liu, D., Wang, D., & Li, H. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2280013 · Cited by 352
An Approximation Theory of Optimal Control for Trainable Manipulators
Saridis, G. N., & Lee, C.-S. G. (1979) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1979.4310171 · Cited by 348
Finite-Approximation-Error-Based Optimal Control Approach for Discrete-Time Nonlinear Systems
Liu, D., & Wei, Q. (2013) · IEEE Transactions on Cybernetics · doi:10.1109/tsmcb.2012.2216523 · Cited by 304
Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles
Vrabie, D., Vamvoudakis, K. G., & Lewis, F. L. (2012) · Institution of Engineering and Technology eBooks · doi:10.1049/pbce081e · Cited by 291
Efficient Exploration in Reinforcement Learning
(2012) · SpringerReference · doi:10.1007/springerreference_178956 · Cited by 257
Off-Policy Temporal Difference Learning with Function Approximation
Precup, D., Sutton, R. S., & Dasgupta, S. (2001) · Cited by 243
Online solution of nonlinear two‐player zero‐sum games using synchronous policy iteration
Vamvoudakis, K. G., & Lewis, F. L. (2011) · International Journal of Robust and Nonlinear Control · doi:10.1002/rnc.1760 · Cited by 238
Abu-Khalaf, M., Lewis, F. L., & Huang, J. (2006) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2006.884959 · Cited by 229
Liu, D., Wang, D., Zhao, D., Wei, Q., & Jin, N. (2012) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2012.2198057 · Cited by 220
Wu, H., & Luo, B. (2012) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2217349 · Cited by 208
Self-citationApproximate Solutions to the Time-Invariant Hamilton–Jacobi–Bellman Equation
Beard, R. W., Saridis, G. N., & Wen, J. T.-Y. (1998) · Journal of Optimization Theory and Applications · doi:10.1023/a:1022664528457 · Cited by 192
Yang, Q., & Jagannathan, S. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2011.2166384 · Cited by 190
Adaptive Learning in Tracking Control Based on the Dual Critic Network Design
Ni, Z., He, H., & Wen, J. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2247627 · Cited by 189
Liu, D., Li, H., & Wang, D. (2013) · Neurocomputing · doi:10.1016/j.neucom.2012.11.021 · Cited by 188
(1972) · Mathematics in Science and Engineering · doi:10.1016/s0076-5392(08)x6031-x · Cited by 184
Toward Off-Policy Learning Control with Function Approximation
Maei, H. R., ri, C. S., Bhatnagar, S., & Sutton, R. (2010) · Cited by 175
A Novel Iterative $\theta $ -Adaptive Dynamic Programming for Discrete-Time Nonlinear Systems
Wei, Q., & Liu, D. (2013) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2013.2280974 · Cited by 138
Adaptive dynamic programming for online solution of a zero-sum differential game
Vrabie, D., & Lewis, F. L. (2011) · Journal of Control Theory and Applications · doi:10.1007/s11768-011-0166-4 · Cited by 136
Modares, H., Lewis, F. L., & Sistani, M. B. N. (2012) · International Journal of Adaptive Control and Signal Processing · doi:10.1002/acs.2348 · Cited by 128
Luo, B., & Wu, H. (2012) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2012.2194781 · Cited by 82
Self-citationLuo, B., & Wu, H. (2012) · International Journal of Robust and Nonlinear Control · doi:10.1002/rnc.2814 · Cited by 66
Self-citationRobust/Optimal Temperature Profile Control of a High-Speed Aerospace Vehicle Using Neural Networks
Yadav, V. H., Padhi, R., & Balakrishnan, S. N. (2007) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.899229 · Cited by 66
Mehraeen, S., Dierks, T., Jagannathan, S., & Crow, M. L. (2012) · IEEE Transactions on Cybernetics · doi:10.1109/tsmcb.2012.2227253 · Cited by 61
Online actor critic algorithm to solve the continuous-time infinite horizon optimal control problem
Vamvoudakis, K. G., & Lewis, F. L. (2009) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2009.5178586 · Cited by 57
Efficient Exploration in Reinforcement Learning
Langford, J. C. (2017) · Springer eBooks · doi:10.1007/978-1-4899-7687-1_244 · Cited by 53
Luo, B., Wu, H., & Li, H. (2014) · Industrial & Engineering Chemistry Research · doi:10.1021/ie4031743 · Cited by 40
Self-citationOnline policy iteration algorithm for optimal control of linear hyperbolic PDE systems
Luo, B., & Wu, H. (2012) · Journal of Process Control · doi:10.1016/j.jprocont.2012.04.011 · Cited by 32
Self-citationData-based approximate policy iteration for nonlinear continuous-time optimal control designPreprint
Luo, B., Wu, H., Huang, T., & Liu, D. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1311.0396 · Cited by 5
Self-citation
Cited by
Optimal and Autonomous Control Using Reinforcement Learning: A Survey
Kiumarsi, B., Vamvoudakis, K. G., Modares, H., & Lewis, F. L. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2773458 · Cited by 891
Adaptive Dynamic Programming for Control: A Survey and Recent Advances
Liu, D., Xue, S., Zhao, B., Luo, B., & Wei, Q. (2021) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2020.3042876 · Cited by 708
Self-citationLiu, D., Yang, X., Wang, D., & Wei, Q. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2417170 · Cited by 440
Adaptive Critic Nonlinear Robust Control: A Survey
Wang, D., He, H., & Liu, D. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2712188 · Cited by 374
Model-Free Optimal Tracking Control via Critic-Only Q-Learning
Luo, B., Liu, D., Huang, T., & Wang, D. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2585520 · Cited by 332
Self-citation
Metadata
- Authors
- Luo, B., Wu, H.-N., & Huang, T.
- Venue
- IEEE Transactions on Cybernetics
- Published
- 2015-01
- Volume
- 45
- Issue
- 1
- Pages
- 65-76
- ISSN
- 2168-2267 (ISSN-L)
- 2168-2275
- OpenAlex
- W2012451615
- Cited by
- 362
- References
- 63
- FWCI
- 29.81
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.