Reinforcement Learning and Approximate Dynamic Programming for Feedback Control
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
In this chapter, we extend the ADP algorithm, dual heuristic programming (DHP), to include a “bootstrapping” parameter λ, analogous to that used in the reinforcement learning algorithm TD(λ). The resulting algorithm, which we call VGL(λ) for value-gradient learning, is proven to produce a weight update that can be equivalent to backpropagation through time (BPTT) applied to a greedy policy on a critic function. This provides a surprising connection between the two alternate methods of BPTT and DHP. Under certain smoothness conditions, VGL(λ=1) with a greedy policy acquires the strong convergence conditions of BPTT, while using a general function approximator for the critic. We show that this can lead to increased stability in the learning of control problems by a neural network
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“ADP represents a kind of data-based control technique which can guarantee the stability of the feedback control system [9].”
From Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach (2023) · cites this paper as [9]
In the citing paper
“Even if an approximated model of the HSA can be developed, the dynamic uncertainty, produced by the mismatch between the approximated model and the true HSA model, will degrade the control performance of the traditionally designed optimal controller [9,12].”
From Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach (2023) · cites this paper as [9]
In the citing paper
“Adaptive dynamic programming (ADP) ensures an e ective way to achieve high performance of the optimal controller which relies on adaptive control, optimal control and reinforcement learning [9, 23-26].”
From Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach (2023) · cites this paper as [9]
Citations
Cites
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem
Vamvoudakis, K. G., & Lewis, F. L. (2010) · Automatica · doi:10.1016/j.automatica.2010.02.018 · Cited by 1,703
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Cited by 1,427
An analysis of temporal-difference learning with function approximation
Tsitsiklis, J. N., & Van Roy, B. (1997) · IEEE Transactions on Automatic Control · doi:10.1109/9.580874 · Cited by 1,194
Prokhorov, D. V., & Wunsch, D. C. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.623201 · Cited by 1,186
Self-citationDiscrete-Time Nonlinear HJB Solution Using Approximate Dynamic Programming: Convergence Proof
Al-Tamimi, A., Lewis, F. L., & Abu-Khalaf, M. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.926614 · Cited by 1,075
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Medical Entomology and Zoology · Cited by 971
Cited by
Items 201–300 of 584
Lv, Y., Zhao, J., Li, R., & Ren, X. (2024) · International Journal of Robust and Nonlinear Control · doi:10.1002/rnc.7191 · Cited by 24
Mix-zero-sum differential games for linear systems with unknown dynamics based on off-policy IRL
Song, R., & Du, K. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.02.078 · Cited by 24
Information Relaxation Bounds for Infinite Horizon Markov Decision Processes
Brown, D. B., & Haugh, M. B. (2017) · Operations Research · doi:10.1287/opre.2017.1631 · Cited by 32
Learning-based adaptive optimal control of linear time-delay systems: A value iteration approach
Cui, L., Pang, B., Krstić, M., & Jiang, Z. (2024) · Automatica · doi:10.1016/j.automatica.2024.111944 · Cited by 23
“Integrating ideas and techniques from RL and control theory, adaptive dynamic programming (ADP) has been developed to conquer the limitations of the conventional RL algorithms (Jiang, Bian, & Gao, 2020; Jiang & Jiang, 2017; Lewis & Liu, 2013).”
Abdulghafoor, A. Z., & Bakolas, E. (2023) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2023.3302288 · Cited by 23
“3], and move to the desired final density p (cid:6) , which is corresponds to a GM that has N T d = 4 components with mean vectors equal to [14 ; 12] , [2 ; 17] , [7 ; 18] , [12 ; 16], covariance matrices equal to 0 .”
BackgroundMethodInfluentialÇimen, M., & Kirkbride, C. (2016) · International Journal of Production Research · doi:10.1080/00207543.2016.1264643 · Cited by 23
“For a broad discussion on the convergence of ADP algorithms in general please refer to, for instance, Lewis and Liu (2013), Nascimento and Powell (2013), Liu, Yang, and Li (2013), Yang, Liu, and Wei (2014), Ni et al. (2015) or Sharaf (2016).”
Value iteration, adaptive dynamic programming, and optimal control of nonlinear systems
Bian, T., & Jiang, Z. (2016) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2016.7798777 · Cited by 24
“See also [16], [21] for two recent review papers, and [19], [30] for two recent books.”
BackgroundA Linearly Relaxed Approximate Linear Program for Markov Decision Processes
Lakshminarayanan, C., Bhatnagar, S., & Szepesvári, C. (2017) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2017.2743163 · Cited by 22
Zhou, Z., Zhou, D., Chen, X., & Shi, X. (2022) · Advances in Space Research · doi:10.1016/j.asr.2022.04.061 · Cited by 22
Feedback Solution to Optimal Switching Problems With Switching Cost
Heydari, A. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2388672 · Cited by 22
Linear optimal tracking control: An adaptive dynamic programming approach
Gao, W., & Jiang, Z. (2015) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.2015.7172106 · Cited by 21
“Approximate/adaptive dynamic programming (ADP) is a non-model-based approach which achieves the online approximation of the cost function; see [2].”
BackgroundZhao, J. (2020) · Neural Processing Letters · doi:10.1007/s11063-020-10220-z · Cited by 21
Zhang, H. (2023) · Journal of Applied Mathematics and Computing · doi:10.1007/s12190-023-01857-9 · Cited by 22
“, Lewis and Liu [21], Jiang and Jiang [14], Kiumarsi et al.”
BackgroundInfluentialGlobal optimality of approximate dynamic programming and its use in non-convex function minimization
Heydari, A., & Balakrishnan, S. N. (2014) · Applied Soft Computing · doi:10.1016/j.asoc.2014.07.003 · Cited by 21
“For an extensive literature on adaptive critic based problems, the reader is referred to [16] and the references in the chapters.”
BackgroundRegular Cycles of Forward and Backward Signal Propagation in Prefrontal Cortex and in Consciousness
Werbos, P. J., & Davis, J. J. J. (2016) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2016.00097 · Cited by 23
“…family have been much more widespread in engineering and computer science, where they have led to major breakthroughs in intelligent control (Lewis and Derong, 2012; Werbos, 2014) and in pattern recognition with “deep learning” (National Science Foundation, 2008; Ng et al, 2008;…”
BackgroundApproximate Optimal Stabilization Control of Servo Mechanisms based on Reinforcement Learning Scheme
Lv, Y., Ren, X., Hu, S., & Xu, H. (2019) · International Journal of Control Automation and Systems · doi:10.1007/s12555-018-0551-6 · Cited by 21
Discrete-Time Multi-Player Games Based on Off-Policy Q-Learning
Li, J., Xiao, Z., & Li, P. (2019) · IEEE Access · doi:10.1109/access.2019.2939384 · Cited by 21
“INTRODUCTION Reinforcement learning (RL), as one of machine learning methods, has been widely used in solving optimal control problems [1]–[4] by using partially or completely unknown dynamics for systems with [5]–[12].”
BackgroundJing, G., Bai, H., George, J., & Chakrabortty, A. (2021) · IEEE Transactions on Control of Network Systems · doi:10.1109/tcns.2021.3074256 · Cited by 21
Multi-machine power system control based on dual heuristic dynamic programming
Ni, Z., Tang, Y., He, H., & Wen, J. (2014) · IEEE Symposium on Computational Intelligence Applications in Smart Grid (CIASG) · doi:10.1109/ciasg.2014.7011566 · Cited by 20
“To date, adaptive dynamic programming (ADP) and reinforcement learning (RL) have been introduced into power system stability and control areas, and have shown the promising performance based on the measured input/output data from the system [1] [2] [3] [4].”
BackgroundPredictive Control with Learning-Based Terminal Costs Using Approximate Value Iteration
Moreno-Mora, F., Beckenbach, L., & Streif, S. (2023) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2023.10.1320 · Cited by 25
Stable Optimal Control and Semicontractive Dynamic Programming
Bertsekas, D. P. (2018) · SIAM Journal on Control and Optimization · doi:10.1137/17m1122815 · Cited by 20
“our problem is often used as the starting point for analysis and algorithmic development; see e.g., the book [VVL13], the papers [JiJ14], [LiW13], the survey papers in the edited volumes [SBP04] and [LeL13], and the special issue [LLL08]. Our purpose here is to consider the problem under weaker conditions and to make…”
MethodDistributed control and optimization of process system networks: A review and perspective
Tang, W., & Daoutidis, P. (2018) · Chinese Journal of Chemical Engineering · doi:10.1016/j.cjche.2018.08.027 · Cited by 22
Gan, M., Zhao, J., & Zhang, C. (2019) · Asian Journal of Control · doi:10.1002/asjc.2243 · Cited by 20
Posterior Sampling-Based Reinforcement Learning for Control of Unknown Linear Systems
Ouyang, Y., Gagrani, M., & Jain, R. (2019) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2019.2950156 · Cited by 28
“Resultsonstochasticadaptivecontrol[5],[6]aresparse.”
Intelligent Trainer for Dyna-Style Model-Based Deep Reinforcement Learning
Dong, L., Li, Y., Zhou, X., Wen, Y., & Guan, K. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3008249 · Cited by 22
Adaptive Online Distributed Optimal Control of Very-Large-Scale Robotic Systems
Zhu, P., Liu, C., & Ferrari, S. (2021) · IEEE Transactions on Control of Network Systems · doi:10.1109/tcns.2021.3097306 · Cited by 23
Neural networks and the experience and cultivation of mind
Werbos, P. J. (2012) · Neural Networks · doi:10.1016/j.neunet.2012.02.026 · Cited by 21
On Improving the Robustness of Reinforcement Learning-based Controllers using Disturbance Observer
Kim, J. W., Shim, H., & Yang, I. (2019) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc40024.2019.9028930 · Cited by 20
“Neural networks and game theory are utilized to address the presence of uncertainties in the nonlinear system by Lewis et al. [1]–[3].”
MethodFast adaptive dynamic programming: Shamanskii iteration for adaptive optimal control
Yang, Y., Gao, W., & Jiang, Z. (2025) · Automatica · doi:10.1016/j.automatica.2025.112494 · Cited by 19
Learning-Based Adaptive Optimal Control for Flotation Processes Subject to Input Constraints
Li, Z., Huang, M., Zhu, J., Gui, W., Jiang, Z., & Du, W. (2022) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2022.3171110 · Cited by 19
Luo, X., Luo, H., & Chang, X. (2015) · International Journal of Distributed Sensor Networks · doi:10.1155/2015/452492 · Cited by 19
Sparse Wide-Area Control of Power Systems using Data-driven Reinforcement Learning
Dizche, A. F., Chakrabortty, A., & Duel‐Hallen, A. (2019) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc.2019.8814445 · Cited by 19
Tan, F. (2020) · Sensors · doi:10.3390/s20051302 · Cited by 20
Model-free adaptive optimal design for trajectory tracking control of rocket-powered vehicle
Nie, W., Li, H., & Zhang, R. (2020) · Chinese Journal of Aeronautics · doi:10.1016/j.cja.2020.02.022 · Cited by 19
Design and Evaluation of a New Machine Learning Framework for IoT and Embedded Devices
Cornetta, G., & Touhafi, A. (2021) · Electronics · doi:10.3390/electronics10050600 · Cited by 20
Wang, J., & Yang, G. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2726004 · Cited by 20
“+ [ 1 0 0 1 ][ ω1k ω2k ]”
BackgroundYang, X., He, H., & Wei, Q. (2018) · International Journal of Robust and Nonlinear Control · doi:10.1002/rnc.4096 · Cited by 19
“20 Due to adaptive critic designs almost sharing the same spirits as reinforcement learning (RL), it is often considered as a synonym for RL. 21 In this paper, we view the adaptive critic approach as a kind of RL.”
BackgroundFarzanegan, B., Moghadam, R., Jagannathan, S., & Pappa, N. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3301383 · Cited by 18
Wang, D., & Mu, C. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.05.051 · Cited by 18
Han, K., Feng, J. Y., & Yao, Y. (2017) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2017.06.017 · Cited by 18
Ballesteros, M., Chaírez, I., & Poznyak, A. S. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.01.016 · Cited by 17
Dong, B., & Li, Y. (2013) · Mathematical Problems in Engineering · doi:10.1155/2013/387817 · Cited by 17
“Reinforcement learning [7, 8] is a kind of learning method mapping situations to actions, so as to maximize a numerical reward signal.”
Li, F., Gao, L., Zhang, Y., & Liu, Y. (2023) · Sustainable Energy Grids and Networks · doi:10.1016/j.segan.2023.101107 · Cited by 17
Cooperative Game-based Multi-Agent Path Planning with Obstacle Avoidance
Guo, Y., Pan, Q., Sun, Q. Y., Zhao, K., Wang, D., & Feng, M. (2019) · IEEE International Symposium on Industrial Electronics (ISIE) · doi:10.1109/isie.2019.8781205 · Cited by 18
Mishra, A., & Ghosh, S. (2022) · Nonlinear Dynamics · doi:10.1007/s11071-021-06908-z · Cited by 17
“The following tuning law has been proposed in [10],[15],[16],[19],[21],[30].”
BackgroundEditorial Special Issue on Adaptive Dynamic Programming and Reinforcement Learning
Liu, D., Lewis, F. L., & Wei, Q. (2020) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2020.3025549 · Cited by 17
From ADP to the brain: Foundations, roadmap, challenges and research priorities
Werbos, P. J. (2014) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2014.6889359 · Cited by 16
“or another. The growth in ADP research in different communities has made it ever more challenging to maintain a unified understanding, which integrates what is being learned in these communities; see [2] for a current survey and effort at integration. The MLCI challenge asks us to come up with the most power possible…”
BackgroundHe, C., Wan, Y., Gu, Y., & Lewis, F. L. (2020) · International Journal of Robust and Nonlinear Control · doi:10.1002/rnc.5122 · Cited by 16
“30 Reinforcement learning (RL) integrates the merits of optimal control and adaptive control to solve optimal control problems online which does not depend on extensive knowledge about the environment.”
BackgroundFast Real-Time Reinforcement Learning for Partially-Observable Large-Scale Systems
Sadamoto, T., & Chakrabortty, A. (2020) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3058228 · Cited by 16
“Since 2000, many non-episodic RL methods have been proposed [10], [16]–[19], including a method called policy iteration [16], [19] that enables fast learning without loss of optimality for (non)linear input affine systems operating around an equilibrium.”
MethodNew insight into the simultaneous policy update algorithms related to H∞ state feedback control
Liu, Z., & Wu, H. (2019) · Information Sciences · doi:10.1016/j.ins.2019.01.060 · Cited by 16
Decentralized Optimal Stabilization of Active Loads in Islanded Microgrids
Dissanayake, A. M., & Ekneligoda, N. C. (2020) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2020.3026361 · Cited by 16
Small-disturbance input-to-state stability of perturbed gradient flows: Applications to LQR problem
Cui, L., Jiang, Z., & Sontag, E. D. (2024) · Systems & Control Letters · doi:10.1016/j.sysconle.2024.105804 · Cited by 15
Analog quantum computing (AQC) and the need for time-symmetric physics
Werbos, P. J., & Dolmatova, L. (2015) · Quantum Information Processing · doi:10.1007/s11128-015-1146-2 · Cited by 15
Stability of infinite-horizon optimal control with discounted cost
Postoyan, R., Buşoniu, L., Nešić, D., & Daafouz, J. (2014) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2014.7039995 · Cited by 15
Machine Learning in Computer Aided Engineering
Montáns, F. J., Cueto, E., & Bathe, K. (2023) · Computational methods in engineering & the sciences · doi:10.1007/978-3-031-36644-4_1 · Cited by 20
Robust Reinforcement Learning for Risk-Sensitive Linear Quadratic Gaussian Control
Cui, L., Başar, T., & Jiang, Z. (2024) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2024.3397928 · Cited by 14
“As in the literature of data-driven control [8], [40], one can satisfy it by means of adding exploration noise, such as sinusoidal signals or random noise.”
BackgroundOptimal Tracking Control of Uncertain Systems
Kiumarsi, B., Modares, H., & Lewis, F. L. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-805246-4.00005-7 · Cited by 15
Neural critic learning toward robust dynamic stabilization
Wang, D., Xu, X., & Zhao, M. (2019) · International Journal of Robust and Nonlinear Control · doi:10.1002/rnc.4860 · Cited by 14
Real-time tracking on adaptive critic design with uniformly ultimately bounded condition
Ni, Z., Fang, X., He, H., Zhao, D., & Xu, X. (2013) · IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning · doi:10.1109/adprl.2013.6614987 · Cited by 14
“We hope to employ the goal network to observe the external reward and provide the critic network with detailed internal signal to fully represent the reward function [15], [4].”
MethodZhu, Z., Wu, W., Chen, T., Hu, J., & Wang, C. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126847 · Cited by 13
de Moura, J. P., Rêgo, P. H. M., & Neto, J. V. D. F. (2019) · ISA Transactions · doi:10.1016/j.isatra.2019.01.010 · Cited by 13
Practical Fast Gradient Sign Attack against Mammographic Image ClassifierPreprint
Yilmaz, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.09610 · Cited by 13
“It is taken in [4].”
Fault detection and isolation of actuator failures in jet engines using adaptive dynamic programming
Kang, H., & Ma, H. (2021) · Applied Mathematics and Computation · doi:10.1016/j.amc.2021.126664 · Cited by 12
Droop-Free Optimal Feedback Control of Distributed Generators in Islanded DC Microgrids
Dissanayake, A. M., & Ekneligoda, N. C. (2019) · IEEE Journal of Emerging and Selected Topics in Power Electronics · doi:10.1109/jestpe.2019.2953869 · Cited by 12
Mishra, A., & Ghosh, S. (2020) · IET Control Theory and Applications · doi:10.1049/iet-cta.2020.0098 · Cited by 12
Energy-saving traffic scheduling in backbone networks with software-defined networks
Lei, J., Deng, S., Lu, Z., He, Y., & Gao, X. (2020) · Cluster Computing · doi:10.1007/s10586-020-03102-5 · Cited by 12
Metamodeling and the Critic-based approach to multi-level optimization
Werbos, L., Kozma, R., Silva, R., Pazienza, G. E., & Werbos, P. J. (2012) · Neural Networks · doi:10.1016/j.neunet.2012.02.036 · Cited by 12
Amirparast, A., & Sani, S. K. H. (2024) · International Journal of Robust and Nonlinear Control · doi:10.1002/rnc.7278 · Cited by 11
Yucelen, T., Sarsılmaz, S. B., & Yildirim, E. (2025) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2025.3554915 · Cited by 11
“For example, robust control [1], [2] and sliding mode control [3], [4] are well-known fixed-gain control approaches, whereas adaptive control [5], [6], [7] and reinforcement learning [8], [9] are well-known adaptive learning approaches.”
Online Optimal Control of Robotic Systems with Single Critic NN‐Based Reinforcement Learning
Long, X., He, Z., & Wang, Z. (2021) · Complexity · doi:10.1155/2021/8839391 · Cited by 11
fault-Tolerant Coverage Control of Multiple Satellites: A Differential Graphical Game Approach
Ni, Y., Yang, H., & Jiang, B. (2023) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2023.3260812 · Cited by 11
Tan, F., & Guan, X. (2018) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2018.2876455 · Cited by 12
Data-Driven Nonzero-Sum Game for Discrete-Time Systems Using Off-Policy Reinforcement Learning
Yang, Y., Zhang, S., Dong, J., & Yin, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2960064 · Cited by 11
“Reinforcement learning (RL) is a bio-inspired methodology concerned with how the agents seek to take actions in an environment to maximize the cumulative reward [21], [22].”
BackgroundElkenawy, A., El-Nagar, A. M., El-Bardini, M. A., & El-Rabaie, N. M. (2021) · Neural Computing and Applications · doi:10.1007/s00521-020-05685-x · Cited by 11
Quantum technology to expand soft computing
Werbos, P. J. (2022) · Systems and Soft Computing · doi:10.1016/j.sasc.2022.200031 · Cited by 11
Rêgo, P. H. M., Neto, J. V. D. F., & Ferreira, E. F. M. (2013) · International Journal of Systems Science · doi:10.1080/00207721.2013.844283 · Cited by 11
“In recent years, those areas have been recognised as promising for the re-alisation of online designs of optimal control, (Lee, Park, and Choi 2010; Vamvoudakis and Lewis 2010; Lewis and Vamvoudakis 2011; Werbos 2012; Liu, Huang, Wang, and Wei 2013).”
BackgroundCheng, Y., Jiang, B., Li, H., & Han, X. (2019) · International Journal of Control Automation and Systems · doi:10.1007/s12555-018-9308-5 · Cited by 11
Model-based and model-free designs for an extended continuous-time LQR with exogenous inputs
Mukherjee, S., Bai, H., & Chakrabortty, A. (2021) · Systems & Control Letters · doi:10.1016/j.sysconle.2021.104983 · Cited by 11
Approximate Dynamic Programming for Trajectory Tracking of Switched Systems
Greene, M. L., Sakha, M. S., Kamalapurkar, R., & Dixon, W. E. (2024) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2024.3453922 · Cited by 10
Drone Networks for Virtual Human Teleportation
Chakareski, J. (2017) · ACM Workshop on Micro Aerial Vehicle Networks, Systems, and Applications (DroNet) · doi:10.1145/3086439.3086448 · Cited by 10
“To examine the efficiency of our algorithm for cooperative drone sensing and control from Section 6, denoted as FEEL (Algorithm 1), we compared it to a state-of-the-art learning method based on adaptive dynamic programming (ADP) [33] and observed that FEEL demonstrates considerable performance gains.”
MethodOnline Learning-based Optimal Control of Nonlinear Systems with Finite-Time Convergence Guarantees
Kokolakis, N.-M. T., & Vamvoudakis, K. G. (2022) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc53348.2022.9867808 · Cited by 10
Data-driven adaptive optimal control for stochastic systems with unmeasurable state
Zhang, M., Gan, M., & Chen, J. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.12.001 · Cited by 10
Approximate Optimal Trajectory Tracking With Sparse Bellman Error Extrapolation
Greene, M. L., Deptula, P., Nivison, S., & Dixon, W. E. (2022) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3194040 · Cited by 12
Zhong, X., Ni, Z., & He, H. (2015) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280471 · Cited by 10
“Adaptive dynamic programming (ADP) has been studied and adopted for seeking the solution of the Hamilton-Jacobi Bellman (HJB) equation over the past decades [1], [2], [3].”
MethodConvergence of Recurrent Neuro-Fuzzy Value-Gradient Learning With and Without an Actor
Al-Dabooni, S., & Wunsch, D. C. (2019) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2019.2912349 · Cited by 10
“In this paper, a gradient of TD( λ ) is combined with ADP to solve the recursive form of the optimal Bellman equation [1] where J ∗ is the optimal value function.”
MethodData-Driven Adaptive Optimal Control for Flotation Processes With Delayed Feedback and Disturbance
Li, Z., Huang, M., Gui, W., & Jiang, Z. (2019) · IEEE Access · doi:10.1109/access.2019.2952396 · Cited by 10
Fitted Q-Function Control Methodology Based on Takagi–Sugeno Systems
Díaz, H., Armesto, L., & Sala, A. (2018) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2018.2885689 · Cited by 10
“1 Policy iteration (PI) and value iteration (VI) are widely used techniques to iteratively compute such optimal value functions and associated policies [7]–[9].”
MethodLiu, C., Zhang, H., Ren, H., & Liang, Y. (2019) · Neural Processing Letters · doi:10.1007/s11063-019-10029-5 · Cited by 10
Q-learning for Optimal Control of Continuous-time SystemsPreprint
Luo, B., Liu, D., & Huang, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.2954 · Cited by 10
“more system information could be extracted for direct control design. In the past few years, the model-free optimal control problem has attracted researchers’ extensive attention in control community [19], [20], and has also brought new challenges to them. Some model-free or partially model-free RL methods [21]– [32]…”
MethodLv, Y., Zhao, J., Zhang, W., & Chang, H. (2024) · Control Theory and Technology · doi:10.1007/s11768-024-00238-2 · Cited by 9
Control Oriented Reinforcement Learning: A Survey of Recent Progress and Applications
Wang, X., Zhang, H., Liu, H., & Lewis, F. L. (2025) · International Journal of Robust and Nonlinear Control · doi:10.1002/rnc.70152 · Cited by 9
Luo, X., & Si, J. (2013) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2013.6707054 · Cited by 9
“It is a modelfree reinforcement learning method with its unique features [14][15] and it has been extensively applied to many complex control problems [5][8][16].”
BackgroundMethodA neuro-adaptive architecture for extremum seeking control using hybrid learning dynamics
Poveda, J. I., Vamvoudakis, K. G., & Benosman, M. (2017) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc.2017.7963009 · Cited by 9
“The design of this type of systems is also motivated by results in the area of neuro-adaptive control [14], and neuro-adaptive optimal control [15], which have been successfully applied in several contexts where the model of the plant is absent or partially known.”
BackgroundKhater, A. A., El-Nagar, A. M., El-Bardini, M. A., & El-Rabaie, N. M. (2019) · European Journal of Control · doi:10.1016/j.ejcon.2019.07.001 · Cited by 9
Han, X., Zhao, X., Wang, D., & Wang, B. (2022) · International Journal of Control · doi:10.1080/00207179.2022.2137852 · Cited by 9
Li, T., Wei, Q., & Wang, F. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3327492 · Cited by 9
“…are several synonyms for ADP, including adaptive critic designs [7], [8], [9], ADP [10], [11], [12], [13], [14], approximate dynamic programming [15], [16], [17], [18], neuro-dynamic programming [19], [20], neural dynamic programming [21], [22], relaxed dynamic programming [23], [24], and…”
BackgroundBoundary Control of 2-D Burgers’ PDE: An Adaptive Dynamic Programming Approach
Talaei, B., Jagannathan, S., & Singler, J. R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2736786 · Cited by 9
“(23) Since the nonlinearity in model is currently estimated, in Section II-B, the ADP control policy will be explained.”
MethodInfluentialZhao, J., & Zhang, C. (2020) · Journal of Industrial and Management Optimization · doi:10.3934/jimo.2020030 · Cited by 9
“Based on equation (6) and Bellman equation (7), we can define the Q-function associated with u k as [15] Substituting system (1) into (8), we can obtain where (9) given by (3), the Q-function (9) have the same value as the Bellman equation (7), that is A detailed proof of Lemma 1 can be found in…”
MethodSecure Blockchain-Enabled DyMonDS Design
Lauer, M., Jaddivada, R., & Ilić, M. (2019) · IEEE International Conference on Omni-layer Intelligent Systems (COINS) · doi:10.1145/3312614.3312654 · Cited by 9
“Together, they are expected to form artificial neural networks with self-adaptive and restorative mechanisms for overall system management [16].”
BackgroundAdaptive sliding mode control of robotic manipulator based on reinforcement learning
Ren, Z., Chen, J., Miao, Y., Miao, Y., Guo, Z., Hu, B., & Lin, R. (2024) · Asian Journal of Control · doi:10.1002/asjc.3363 · Cited by 8
Metadata
- Authors
- Fairbank, M., Prokhorov, D., & Alonso, E.
- Affiliations
- The University of Texas at Arlington (US)
- University of Illinois Chicago (US)
- Venue
- Wiley eBooks
- Publisher
- Wiley
- Published
- 2012-12-17
- OpenAlex
- W560518094
- Cited by
- 577
- References
- 33
- Citations by year
- 2026: 15 · 2025: 27 · 2024: 38 · 2023: 48 · 2022: 38 · 2021: 50 · 2020: 44 · 2019: 64 · 2018: 35 · 2017: 54 · 2016: 43 · 2015: 39 · 2014: 48 · 2013: 27 · 2012: 7
- FWCI
- 6.42
- Type
- book
- Language
- en
- Open access
- closed
- Licence
- doi.wiley.com/10.1002/tdm_license_1.1 (tdm)
- Locations
- Wiley eBooks (publishedVersion)
- City Research Online (City University London) (acceptedVersion)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Advanced Control Systems Optimization
- Keywords
- reinforcement learning, approximate dynamic programming, feedback control, adaptive dynamic programming, multiplayer games, decision-making, robot control, industrial process control, economic decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.