Reinforcement Learning and Approximate Dynamic Programming for Feedback Control
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
In this chapter, we extend the ADP algorithm, dual heuristic programming (DHP), to include a “bootstrapping” parameter λ, analogous to that used in the reinforcement learning algorithm TD(λ). The resulting algorithm, which we call VGL(λ) for value-gradient learning, is proven to produce a weight update that can be equivalent to backpropagation through time (BPTT) applied to a greedy policy on a critic function. This provides a surprising connection between the two alternate methods of BPTT and DHP. Under certain smoothness conditions, VGL(λ=1) with a greedy policy acquires the strong convergence conditions of BPTT, while using a general function approximator for the critic. We show that this can lead to increased stability in the learning of control problems by a neural network
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“ADP represents a kind of data-based control technique which can guarantee the stability of the feedback control system [9].”
From Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach (2023) · cites this paper as [9]
In the citing paper
“Even if an approximated model of the HSA can be developed, the dynamic uncertainty, produced by the mismatch between the approximated model and the true HSA model, will degrade the control performance of the traditionally designed optimal controller [9,12].”
From Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach (2023) · cites this paper as [9]
In the citing paper
“Adaptive dynamic programming (ADP) ensures an e ective way to achieve high performance of the optimal controller which relies on adaptive control, optimal control and reinforcement learning [9, 23-26].”
From Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach (2023) · cites this paper as [9]
Citations
Cites
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem
Vamvoudakis, K. G., & Lewis, F. L. (2010) · Automatica · doi:10.1016/j.automatica.2010.02.018 · Cited by 1,703
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Cited by 1,427
An analysis of temporal-difference learning with function approximation
Tsitsiklis, J. N., & Van Roy, B. (1997) · IEEE Transactions on Automatic Control · doi:10.1109/9.580874 · Cited by 1,194
Prokhorov, D. V., & Wunsch, D. C. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.623201 · Cited by 1,186
Self-citationDiscrete-Time Nonlinear HJB Solution Using Approximate Dynamic Programming: Convergence Proof
Al-Tamimi, A., Lewis, F. L., & Abu-Khalaf, M. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.926614 · Cited by 1,075
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Medical Entomology and Zoology · Cited by 971
Cited by
Adaptive Dynamic Programming for Control: A Survey and Recent Advances
Liu, D., Xue, S., Zhao, B., Luo, B., & Wei, Q. (2021) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2020.3042876 · Cited by 708
Modares, H., & Lewis, F. L. (2014) · Automatica · doi:10.1016/j.automatica.2014.05.011 · Cited by 701
“Strong connections between RL and optimal control have prompted a major effort towards introducing and developing online and model-free RL algorithms to learn the solution to optimal control problems (Lewis & Liu, 2012; Vrabie, Vamvoudakis, & Lewis, 2013; Zhang, Liu, Luo, & Wang, 2012).”
BackgroundKiumarsi, B., Lewis, F. L., Modares, H., Karimpour, A., & Naghibi-Sistani, M.-B. (2014) · Automatica · doi:10.1016/j.automatica.2014.02.015 · Cited by 574
Liu, D., Yang, X., Wang, D., & Wei, Q. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2417170 · Cited by 440
“By utilizing the gradient descent algorithm, the weight tuning law for the critic NN is generally given as [31]–[34], [40], and [41]”
BackgroundMethodInfluentialAdaptive Dynamic Programming and Adaptive Optimal Output Regulation of Linear Systems
Gao, W., & Jiang, Z.-P. (2016) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2016.2548662 · Cited by 413
“, [10], [12], [19]–[21], [23], [24], [26], [30], [31], [33], [35].”
Universal quantum control through deep reinforcement learning
Niu, M. Y., Boixo, S., Smelyanskiy, V. N., & Neven, H. (2019) · npj Quantum Information · doi:10.1038/s41534-019-0141-3 · Cited by 381
Robust Adaptive Dynamic Programming and Feedback Stabilization of Nonlinear Systems
Jiang, Y., & Jiang, Z.-P. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2294968 · Cited by 414
Lee, J. H., Shin, J., & Realff, M. J. (2018) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2017.10.008 · Cited by 387
Wang, D., Gao, N., Liu, D., Li, J., & Lewis, F. L. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123843 · Cited by 364
Adaptive Critic Nonlinear Robust Control: A Survey
Wang, D., He, H., & Liu, D. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2712188 · Cited by 374
“[87]–[89], the ADP technique is closely related to reinforcement learning”
BackgroundMethodLiu, D., Wang, D., & Li, H. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2280013 · Cited by 352
“There are several synonyms used for ADP, including adaptive dynamic programming [11]–[15], approximate dynamic programming [16]–[18], neuro-dynamic programming [19], neural dynamic programming [20], adaptive critic designs [21], and reinforcement learning [22].”
MethodSolving the optimal path planning of a mobile robot using improved Q-learning
Low, E. S., Ong, P., & Cheah, K. C. (2019) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2019.02.013 · Cited by 326
Actor–Critic-Based Optimal Tracking for Partially Unknown Nonlinear Discrete-Time Systems
Kiumarsi, B., & Lewis, F. L. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2358227 · Cited by 326
Model-Free Optimal Tracking Control via Critic-Only Q-Learning
Luo, B., Liu, D., Huang, T., & Wang, D. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2585520 · Cited by 332
Air-Breathing Hypersonic Vehicle Tracking Control Based on Adaptive Dynamic Programming
Mu, C., Ni, Z., Sun, C., & He, H. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2516948 · Cited by 307
Peng, Z., Hu, J., Shi, K., Luo, R., Huang, R., Ghosh, B. K., & Huang, J. (2020) · Applied Mathematics and Computation · doi:10.1016/j.amc.2019.124821 · Cited by 302
Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learning
Li, Y., Wen, Y., Tao, D., & Guan, K. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2927410 · Cited by 274
“Reinforcement Learning [11] [30] deals with agents that learn to take better actions directly from experience of interacting with the environment.”
BackgroundData-based approximate policy iteration for affine nonlinear continuous-time optimal control design
Luo, B., Wu, H.-N., Huang, T., & Liu, D. (2014) · Automatica · doi:10.1016/j.automatica.2014.10.056 · Cited by 280
An Event-Triggered ADP Control Approach for Continuous-Time System With Unknown Internal States
Zhong, X., & He, H. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2523878 · Cited by 263
“It has been widely recognized as one of the “core methodologies” to achieve the optimal control for intelligent systems in a general case [5]–[8].”
BackgroundOptimal Tracking Control of Unknown Discrete-Time Linear Systems Using Input-Output Measured Data
Kiumarsi, B., Lewis, F. L., Naghibi-Sistani, M.-B., & Karimpour, A. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2384016 · Cited by 249
“Instead of directly solving for the ARE, PI methods start with an admissible control policy and solve a sequence of Lyapunov equations to find the optimal control policy [18].”
MethodValue and Policy Iterations in Optimal Control and Adaptive Dynamic Programming
Bertsekas, D. P. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2503980 · Cited by 238
“For a selective list of recent references, which themselves contain extensive lists of other references, see the book [20], the papers [25]–[29], the survey papers in the edited volumes [30] and [31], and the special issue [24].”
Li, H., Liu, D., & Wang, D. (2014) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2014.2300532 · Cited by 231
1 Global Adaptive Dynamic Programming for Continuous-Time Nonlinear SystemsPreprint
Jiang, Y., Member, S., & Jiang, Z. (2016) · Cited by 235
Gao, W., Jiang, Y., Jiang, Z.-P., & Chai, T. (2016) · Automatica · doi:10.1016/j.automatica.2016.05.008 · Cited by 234
“Approximate/adaptive dynamic programming (ADP) (see, e.g., Lewis & Liu, 2013;Ni, He, &Wen, 2013; Si, Barto, Powell, &Wunsch, 2004; Vamvoudakis, 2014; Werbos, 1974, 1990; Zhang, Liu, Luo, & Wang, 2013) is a non-model-based approach which gives rise to online approximation of optimal solutions via…”
Zhang, H., Qin, C., & Luo, Y. (2014) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2014.2303139 · Cited by 234
Experience Replay for Optimal Control of Nonzero-Sum Game Systems With Unknown Dynamics
Zhao, D., Zhang, Q., Wang, D., & Zhu, Y. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2488680 · Cited by 229
Wei, Q., & Liu, D. (2014) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2013.2284545 · Cited by 224
Data-Based Adaptive Critic Designs for Nonlinear Robust Optimal Control With Uncertain Dynamics
Wang, D., Liu, D., Zhang, Q., & Zhao, D. (2016) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2015.2492941 · Cited by 218
“The comprehensive research progress and prospects of ADP for optimal control can be found in [8] and [9].”
BackgroundVamvoudakis, K. G., Miranda, M. F., & Hespanha, J. P. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2487972 · Cited by 210
“But all the existing algorithms — such as the ones developed in [6–10] and most of the references therein — can only guarantee uniform ultimate boundedness of the closedloop system, i.”
MethodA deep reinforcement learning approach for chemical production scheduling
Hubbs, C. D., Li, C., Sahinidis, N. V., Grossmann, I. E., & Wassick, J. M. (2020) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2020.106982 · Cited by 196
Yang, X., Liu, D., & Wang, D. (2014) · International Journal of Control · doi:10.1080/00207179.2013.848292 · Cited by 203
“It aims to find the stabilising con-trol to maximise or minimise a specified performance index (Kek, Teo, & Ismail, 2010; Koshkouei, Farahi, & Burnham, 2012; Lewis & Liu, 2013; Lewis & Syrmos, 1995; Sahnoun, Andrieu, & Nadri, 2012; Xin & Pan, 2010; Zhang, Liu, Luo, & Wang, 2013).”
Intelligent Train Operation Algorithms for Subway by Expert System and Reinforcement Learning
Yin, J., Chen, D., & Li, L. (2014) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2014.2320757 · Cited by 198
(2017) · IEEE Control Systems · doi:10.1109/mcs.2016.2621461 · Cited by 192
The intelligent critic framework for advanced optimal control
Wang, D., Ha, M., & Zhao, M. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10118-9 · Cited by 192
Hamiltonian-Driven Adaptive Dynamic Programming With Efficient Experience Replay
Yang, Y., Pan, Y., Xu, C.-Z., & Wunsch, D. C. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3213566 · Cited by 188
Adaptive Learning in Tracking Control Based on the Dual Critic Network Design
Ni, Z., He, H., & Wen, J. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2247627 · Cited by 189
Mu, C., Ni, Z., Sun, C., & He, H. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2548941 · Cited by 181
“extended to a new level by the adaptive dynamic programming (ADP) method [23]–[25].”
MethodEvent-Triggered Optimal Control With Performance Guarantees Using Adaptive Dynamic Programming
Luo, B., Yang, Y., Liu, D., & Wu, H.-N. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2899594 · Cited by 177
Wei, Q., & Liu, D. (2014) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2014.2301770 · Cited by 179
Robust adaptive dynamic programming for linear and nonlinear systems: An overview
Jiang, Z.-P., & Jiang, Y. (2013) · European Journal of Control · doi:10.1016/j.ejcon.2013.05.017 · Cited by 171
On Mixed Data and Event Driven Design for Adaptive-Critic-Based Nonlinear $H_{\infty}$ Control
Wang, D., Mu, C., Liu, D., & Ma, H. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2642128 · Cited by 168
“The comprehensive research progress and prospects of ADP-based optimal control have been reported in [7] and [8].”
BackgroundData-Driven Flotation Industrial Process Operational Optimal Control Based on Reinforcement Learning
Jiang, Y., Fan, J., Chai, T., Li, J., & Lewis, F. L. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2017.2761852 · Cited by 163
“This paper develops a model-free data-driven method for OOC for a single-cell flotation industrial process, which is based on reinforcement learning (RL) [19]–[27].”
MethodRevealing cascading failure vulnerability in power grids using risk-graph
Zhu, Y., Yan, J., Sun, Y., & He, H. (2014) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2013.2295814 · Cited by 161
“For instance, dynamic programming (DP) [28] and approximate dynamic programming (ADP) [29], [30] are the popular techniques in solving the problems that have the properties of overlapping subproblems.”
BackgroundWang, D., Liu, D., Li, H., & Ma, H. (2014) · Information Sciences · doi:10.1016/j.ins.2014.05.050 · Cited by 159
Jiang, Y., & Jiang, Z. (2012) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2012.2213353 · Cited by 154
Zhong, X., He, H., Zhang, H., & Wang, Z. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2305841 · Cited by 153
“Here, we highlight a few important ADP researches from the theoretical perspective that are closely related to the research presented in this paper, and interested readers can refer to the two important handbooks on ADP for many other successful architectures, algorithms, models, and challenging engineering applicatio…”
Event-Triggered ADP for Tracking Control of Partially Unknown Constrained Uncertain Systems
Xue, S., Luo, B., Liu, D., & Gao, Y. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3054626 · Cited by 153
“Generally speaking, reinforcement learning methods aim to learn the optimal policy with the interaction between an agent and the environment [2], [3].”
BackgroundOff-Policy Interleaved $Q$ -Learning: Optimal Control for Affine Nonlinear Discrete-Time Systems
Li, J., Chai, T., Lewis, F. L., Ding, Z., & Jiang, Y. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2861945 · Cited by 146
“REINFORCEMENT learning (RL), one of machine learning tools, has become a powerful and practical tool for tackling optimal control problems [1]–[4].”
BackgroundGame-Theoretic Approach to Feedback-Driven Multi-stage Moving Target Defense
Zhu, Q., & Başar, T. (2013) · Lecture notes in computer science · doi:10.1007/978-3-319-02786-9_15 · Cited by 145
Cooperative Finitely Excited Learning for Dynamical Games
Yang, Y., Modares, H., Vamvoudakis, K. G., & Lewis, F. L. (2023) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2023.3274908 · Cited by 143
“Reinforcement learning (RL) and adaptive dynamic programming (ADP) bring together adaptive critic design [18] with dynamic programming [19] to assist the agent in optimizing a long-term reward through interaction with the environment [20].”
BackgroundComplete stability analysis of a heuristic approximate dynamic programming control design
Sokolov, Y., Kozma, R., Werbos, L., & Werbos, P. J. (2015) · Automatica · doi:10.1016/j.automatica.2015.06.001 · Cited by 141
“We also have internal iterations, which are not explicit (Lewis & Liu, 2012; He 2011).”
InfluentialLuo, B., Wu, H., & Li, H. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2320744 · Cited by 141
“In recent years, some NDP approaches have been introduced to synthesize controller for ODE systems [25]–[27].”
BackgroundFeature-based aggregation and deep reinforcement learning: a survey and some new implementations
Bertsekas, D. P. (2018) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2018.7511249 · Cited by 159
Fault-tolerant control of a hydraulic servo actuator via adaptive dynamic programming
Stojanović, V. (2023) · Mathematical Modelling and Control · doi:10.3934/mmc.2023016 · Cited by 141
Zhao, B., Liu, D., & Li, Y. (2016) · Information Sciences · doi:10.1016/j.ins.2016.12.016 · Cited by 132
Rizvi, S. A. A., & Lin, Z. (2018) · Automatica · doi:10.1016/j.automatica.2018.05.027 · Cited by 131
A novel adaptive dynamic programming based on tracking error for nonlinear discrete-time systems
Li, C., Ding, J., Lewis, F. L., & Chai, T. (2021) · Automatica · doi:10.1016/j.automatica.2021.109687 · Cited by 131
Yang, X., He, H., & Liu, D. (2017) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2017.2774602 · Cited by 132
“Since then, there were plenty of ADP approaches developed in [16]–[22].”
MethodInfluentialAdaptive Optimal Control of Heterogeneous CACC System With Uncertain Dynamics
Zhu, Y., Zhao, D., & Zhong, Z. (2018) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2018.2811376 · Cited by 129
“Adaptive dynamic programming (ADP) is capable of solving optimal control problems without dynamics information (see [14], [15] and references cited therein).”
BackgroundModel-Free Adaptive Control for Unknown Nonlinear Zero-Sum Differential Game
Zhong, X., He, H., Wang, D., & Ni, Z. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2712617 · Cited by 128
“So far, ADP method has been recognized as one of the core methodologies to achieve optimization for intelligent systems, especially the nonlinear systems [17]–[22].”
MethodJiang, Y., Fan, J., Chai, T., Lewis, F. L., & Li, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2771459 · Cited by 133
Li, S., Won, H., Fu, X., Fairbank, M., Wunsch, D. C., & Alonso, E. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2897653 · Cited by 121
Self-citationLi, G., & Görges, D. (2018) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2018.2877389 · Cited by 120
“Widely applied value function approximations in practical applications are neural networks, linear basis functions and piecewise linear functions [28].”
BackgroundDong, W., Li, S., Fu, X., Li, Z., Fairbank, M., & Gao, Y. (2021) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2021.3053468 · Cited by 119
“The typical structure of discrete-time ADP includes a discrete-time state-space system model and a performance index or cost associated with the system [17], [18].”
BackgroundSelf-citationModel-Free Adaptive Optimal Control for Unknown Nonlinear Multiplayer Nonzero-Sum Game
Wei, Q., Zhu, L., Song, R., Zhang, P., Liu, D., & Xiao, J. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3030127 · Cited by 117
Stability Analysis of Discrete-Time Infinite-Horizon Optimal Control With Discounted Cost
Postoyan, R., Buşoniu, L., Nešić, D., & Daafouz, J. (2016) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2016.2616644 · Cited by 116
The One-Dimensional Dynamic Dispatch Waves Problem
Klapp, M. A., Erera, A. L., & Toriello, A. (2016) · Transportation Science · doi:10.1287/trsc.2016.0682 · Cited by 113
“At any dispatch epoch (wave), the information available to the decision maker is (1) a set of known, open requests which remain unfulfilled, and (2) a set of potential requests that may arrive later in the service day.”
BackgroundEnergy-Storage-Based Intelligent Frequency Control of Microgrid With Stochastic Model Uncertainties
Mu, C., Zhang, Y., Jia, H., & He, H. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2942770 · Cited by 113
Ding, L., Li, S., Gao, H., Liu, Y., Huang, L., & Deng, Z. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2939424 · Cited by 115
Qin, C., Zhang, H., & Luo, Y. (2013) · International Journal of Control · doi:10.1080/00207179.2013.863432 · Cited by 112
“…recent years, adaptive dynamic programming (ADP) has been studied by many researchers to solve optimal control problems for uncertain systems (Lewis & Liu, 2012; Lewis, Vrabie, & Vamvoudakis, 2012; Liu & Wei, 2013; Prokhorov & Wunsch, 1997; Vamvoudakis & Lewis, 2010; Wang et al., 2009, 2012;…”
BackgroundNear-Optimal Tracking Control of Mobile Robots Via Receding-Horizon Dual Heuristic Programming
Lian, C., Xu, X., Chen, H., & He, H. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2478857 · Cited by 111
“Index Terms—Approximate dynamic programming (ADP), model predictive control (MPC), receding horizon, reinforcement learning, trajectory tracking, wheeled mobile robots (WMRs).”
BackgroundInfluentialYang, X., & He, H. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2946122 · Cited by 111
“To overcome the bottleneck, adaptive dynamic programming (ADP) [9] and reinforcement learning (RL) [10] were introduced, which aimed at obtaining the numerical solutions of HJBEs.”
MethodParameterized Batch Reinforcement Learning for Longitudinal Control of Autonomous Land Vehicles
Huang, Z., Xu, X., He, H., Tan, J. H., & Sun, Z. (2017) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2017.2712561 · Cited by 109
“More details about actor-critic algorithms can be referred to [43].”
BackgroundMethodInfluentialModel-Free Dual Heuristic Dynamic Programming
Ni, Z., He, H., Zhong, X., & Prokhorov, D. V. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2424971 · Cited by 108
“Adaptive dynamic programming (ADP) has demonstrated great potential in solving the optimal control problems in a principled way in [1]–[4].”
BackgroundSelf-citationModel‐based vs data‐driven adaptive control: An overview
Benosman, M. (2018) · International Journal of Adaptive Control and Signal Processing · doi:10.1002/acs.2862 · Cited by 119
Djordjevic, V., Tao, H., Song, X., He, S., Gao, W., & Stojanovic, V. (2023) · Mathematical Biosciences and Engineering · doi:10.3934/mbe.2023376 · Cited by 107
Goal Representation Heuristic Dynamic Programming on Maze Navigation
Ni, Z., He, H., Wen, J., & Xu, X. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2271454 · Cited by 106
Wang, D., He, H., Mu, C., & Liu, D. (2017) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2017.2674633 · Cited by 105
“Therein, function approximation structures, such as artificial neural networks, were always included [17], [18].”
BackgroundLearning and Guaranteed Cost Control With Event-Based Adaptive Critic Implementation
Wang, D., & Liu, D. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2817256 · Cited by 103
“For overcoming the difficulty (caused by “curse of dimensionality”) in coping with the HJB equation, based on function approximation structures like artificial neural networks, the adaptive or approximate dynamic programming (ADP) methodology was developed by Werbos [1] to solve optimal control problems effectively an…”
MethodWu, N., & Wang, H. (2018) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2018.09.052 · Cited by 100
Robust Policy Iteration for Continuous-Time Linear Quadratic Regulation
Pang, B., Bian, T., & Jiang, Z. (2021) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2021.3085510 · Cited by 97
Hamiltonian-Driven Adaptive Dynamic Programming for Continuous Nonlinear Dynamical Systems
Yang, Y., Wunsch, D. C., & Yin, Y. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2654324 · Cited by 96
“It should be noted that this definition of the Hamiltonian is slightly different from those found elsewhere, such as [8], [9], [11], [28], and [36].”
ResultZhao, B., Liu, D., & Alippi, C. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2962011 · Cited by 95
Zhang, H., Liu, C., Su, H., & Zhang, K. (2020) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2019.2958484 · Cited by 95
Event-Triggered Output Feedback Control of Switched Nonlinear Systems With Input Saturation
Li, H., Zhang, X., & Feng, G. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2965142 · Cited by 94
Data-driven control of hydraulic servo actuator based on adaptive dynamic programming
Djordjevic, V., Stojanović, V., Tao, H., Song, X., He, S., & Gao, W. (2022) · Discrete and Continuous Dynamical Systems - S · doi:10.3934/dcdss.2021145 · Cited by 98
Adaptive Dynamic Programming for Stochastic Systems With State and Control Dependent Noise
Bian, T., Jiang, Y., & Jiang, Z. (2016) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2016.2550518 · Cited by 93
“See also [20], [27], [28] for review papers, and [25], [41] for two recent books.”
BackgroundPeng, Z., Zhao, Y., Hu, J., & Ghosh, B. K. (2018) · Information Sciences · doi:10.1016/j.ins.2018.12.079 · Cited by 91
GrDHP: A General Utility Function Representation for Dual Heuristic Dynamic Programming
Ni, Z., He, H., Zhao, D., Xu, X., & Prokhorov, D. V. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2329942 · Cited by 90
Self-citationA Survey on Applications of Model-Free Strategy Learning in Cognitive Wireless Networks
Wang, W., Kwasinski, A., Niyato, D., & Han, Z. (2016) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2016.2539923 · Cited by 90
Tang, W., Wang, Y., Jiao, X., & Ren, L. (2022) · Energy · doi:10.1016/j.energy.2022.126264 · Cited by 90
Zhang, X., Liu, J., Xu, X., Yu, S., & Chen, H. (2022) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2022.3146284 · Cited by 89
Yang, X., & He, H. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.11.022 · Cited by 86
Zhu, Y., Zhao, D., & Li, X. (2016) · IET Control Theory and Applications · doi:10.1049/iet-cta.2015.0769 · Cited by 85
Hamiltonian-Driven Hybrid Adaptive Dynamic Programming
Yang, Y., Vamvoudakis, K. G., Modares, H., Yin, Y., & Wunsch, D. C. (2020) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2019.2962103 · Cited by 84
“Reinforcement learning (RL) and/or ADP, which evolves from off-line iteration [9]–[12] to online learning [13], [14], have been successfully developed to provide an efficient way to obtain optimal controllers in an adaptive manner online [15], [16].”
BackgroundYang, X., Liu, D., Luo, B., & Li, C. (2016) · Information Sciences · doi:10.1016/j.ins.2016.07.051 · Cited by 84
UAV-IoT for Next Generation Virtual Reality
Chakareski, J. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2921869 · Cited by 83
“ADP (approximate dynamic programming) [72], a state-ofthe-art learning method, and USP (uniform spatial placement), a traditional UAV coverage method [73].”
MethodDing, J., Modares, H., Chai, T., & Lewis, F. L. (2016) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2016.2516973 · Cited by 83
“In the following, we provide a method to estimate these functions online based on value function approximation (VFI) [11], [14], [15] where a parameterized form is assumed for the value functions in (1).”
MethodA New Powered Lower Limb Prosthesis Control Framework Based on Adaptive Dynamic Programming
Wen, Y., Si, J., Gao, X. H., Huang, S., & Huang, H. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2584559 · Cited by 82
“In recent years, it has emerged as a powerful alternative to classic optimal control, especially for complex nonlinear systems, due to its demonstrated scalability for large systems and flexibility and capability of learning from data measurements [1], [2].”
BackgroundAdaptive Dynamic Programming for Robust Regulation and Its Application to Power Systems
Yang, X., He, H., & Zhong, X. (2017) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2017.2782205 · Cited by 80
Metadata
- Authors
- Fairbank, M., Prokhorov, D., & Alonso, E.
- Affiliations
- The University of Texas at Arlington (US)
- University of Illinois Chicago (US)
- Venue
- Wiley eBooks
- Publisher
- Wiley
- Published
- 2012-12-17
- OpenAlex
- W560518094
- Cited by
- 577
- References
- 33
- Citations by year
- 2026: 15 · 2025: 27 · 2024: 38 · 2023: 48 · 2022: 38 · 2021: 50 · 2020: 44 · 2019: 64 · 2018: 35 · 2017: 54 · 2016: 43 · 2015: 39 · 2014: 48 · 2013: 27 · 2012: 7
- FWCI
- 6.42
- Type
- book
- Language
- en
- Open access
- closed
- Licence
- doi.wiley.com/10.1002/tdm_license_1.1 (tdm)
- Locations
- Wiley eBooks (publishedVersion)
- City Research Online (City University London) (acceptedVersion)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Advanced Control Systems Optimization
- Keywords
- reinforcement learning, approximate dynamic programming, feedback control, adaptive dynamic programming, multiplayer games, decision-making, robot control, industrial process control, economic decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.