Reinforcement Learning and Approximate Dynamic Programming for Feedback Control
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
In this chapter, we extend the ADP algorithm, dual heuristic programming (DHP), to include a “bootstrapping” parameter λ, analogous to that used in the reinforcement learning algorithm TD(λ). The resulting algorithm, which we call VGL(λ) for value-gradient learning, is proven to produce a weight update that can be equivalent to backpropagation through time (BPTT) applied to a greedy policy on a critic function. This provides a surprising connection between the two alternate methods of BPTT and DHP. Under certain smoothness conditions, VGL(λ=1) with a greedy policy acquires the strong convergence conditions of BPTT, while using a general function approximator for the critic. We show that this can lead to increased stability in the learning of control problems by a neural network
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“ADP represents a kind of data-based control technique which can guarantee the stability of the feedback control system [9].”
From Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach (2023) · cites this paper as [9]
In the citing paper
“Even if an approximated model of the HSA can be developed, the dynamic uncertainty, produced by the mismatch between the approximated model and the true HSA model, will degrade the control performance of the traditionally designed optimal controller [9,12].”
From Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach (2023) · cites this paper as [9]
In the citing paper
“Adaptive dynamic programming (ADP) ensures an e ective way to achieve high performance of the optimal controller which relies on adaptive control, optimal control and reinforcement learning [9, 23-26].”
From Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach (2023) · cites this paper as [9]
Citations
Cites
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem
Vamvoudakis, K. G., & Lewis, F. L. (2010) · Automatica · doi:10.1016/j.automatica.2010.02.018 · Cited by 1,703
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Cited by 1,427
An analysis of temporal-difference learning with function approximation
Tsitsiklis, J. N., & Van Roy, B. (1997) · IEEE Transactions on Automatic Control · doi:10.1109/9.580874 · Cited by 1,194
Prokhorov, D. V., & Wunsch, D. C. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.623201 · Cited by 1,186
Self-citationDiscrete-Time Nonlinear HJB Solution Using Approximate Dynamic Programming: Convergence Proof
Al-Tamimi, A., Lewis, F. L., & Abu-Khalaf, M. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.926614 · Cited by 1,075
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Medical Entomology and Zoology · Cited by 971
Cited by
Items 301–400 of 584
A personalized reinforcement learning recommendation algorithm using bi-clustering techniques
Waqar, M., & Ayub, M. (2025) · PLoS ONE · doi:10.1371/journal.pone.0315533 · Cited by 8
Kokolakis, N.-M. T., Zhang, Z., Liu, S., Vamvoudakis, K. G., Darbon, J., & Karniadakis, G. E. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3554713 · Cited by 8
Composite Adaptive Critic Design
Yang, Y., Modares, H., Vamvoudakis, K. G., & Lewis, F. L. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2025.125435 · Cited by 8
You, S., Byeon, K., Seo, J., Kim, W., & Tomizuka, M. (2025) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2025.3532518 · Cited by 8
“Reinforcement learning (RL) has attracted considerable attention across various fields, including control systems for its ability to solve sophisticated and hard-to-model optimization problems [27], [28], [29], [30].”
Talaei, B., Xu, H., & Jagannathan, S. (2014) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2014.7040453 · Cited by 8
“Under the assumption that V (x̄)∈C1, an infinitesimal equivalent to (12) is given by [13]”
BackgroundProximal algorithms and temporal difference methods for solving fixed point problems
Bertsekas, D. P. (2018) · Computational Optimization and Applications · doi:10.1007/s10589-018-9990-5 · Cited by 8
“[7], Sutton and Barto [57], Si et al. [50], Powell [48], Busoniu et al. [1], Szepesvari [59], Bertsekas [17], Lewis and Liu [42], and Vrabie, Vamvoudakis, and Lewis [63].”
Mazouchi, M., Nageshrao, S. P., & Modares, H. (2023) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2023.3245824 · Cited by 11
Probabilistic control and majorisation of optimal control
Lefebvre, T. (2024) · Systems & Control Letters · doi:10.1016/j.sysconle.2024.105837 · Cited by 10
Concurrent learning-based network synchronization
Klotz, J. R., Kamalapurkar, R., & Dixon, W. E. (2014) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.2014.6859099 · Cited by 8
Yang, X., Liu, D., Wang, D., & Ma, H. (2014) · Chinese Control Conference (CCC) · doi:10.1109/chicc.2014.6896465 · Cited by 8
Approximate solution to discrete-time optimal switching problems with minimum dwell time constraint
Heydari, A. (2016) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.2016.7524996 · Cited by 8
“Such switching problems were investigated in [19], where an ADP based method was developed that determines the switching times beforehand through solving the minimization of an algebraic equation based on the initial conditions.”
MethodInfluentialOnline Learning Algorithm Based on Adaptive Control Theory
Liu, J., Zhou, J., Kamel, M. S., & Luo, X.-L. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2688413 · Cited by 8
“There are several synonyms used for ADP, including adaptive dynamic programming [28]–[32], approximate dynamic programming [33], [34], neural dynamic programming [35]–[37], adaptive critic designs [38]–[41], and reinforcement learning [42].”
MethodNear optimal control and network co-design for uncertain networked control system with constraints
Xu, H., & Carrillo, L. R. G. (2017) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc.2017.7963302 · Cited by 8
A Least-Squares Temporal Difference based method for solving resource allocation problems
Forootani, A., Tipaldi, M., Zarch, M. G., Liuzza, D., & Glielmo, L. (2020) · IFAC Journal of Systems and Control · doi:10.1016/j.ifacsc.2020.100106 · Cited by 8
“This is due to the fact that, though the projected Bellman operator can be a contraction, it lacks of monotonicity (Bertsekas, 2012a), paragraph 6.4.3.”
BackgroundMethodInfluentialValue‐iteration‐based affine nonlinear optimal control involving admissibility discussion
Wang, D., Ren, J., & Ha, M. (2022) · International Journal of Robust and Nonlinear Control · doi:10.1002/rnc.6213 · Cited by 8
“The iterative ADP method is an effective technique to iteratively approximate the optimal solution of (4).”
MethodThe Dynamic Programming Principle
Brandimarte, P. (2021) · EURO advanced tutorials on operational research · doi:10.1007/978-3-030-61867-4_1 · Cited by 8
Online Partially Model-Free Solution of Two-Player Zero Sum Differential Games
Praveen, P., & Bhasin, S. (2013) · IFAC Proceedings Volumes · doi:10.3182/20131218-3-in-2045.00142 · Cited by 7
A tractor-trailer parking control scheme using adaptive dynamic programming
Guan, C., & Jiang, Y. (2021) · Complex & Intelligent Systems · doi:10.1007/s40747-021-00330-z · Cited by 7
“Stability issues in ADP-based control systems design are addressed in [3,20,38].”
BackgroundQin, Z., Zhu, H. S., Wang, S., Xin, Y., & Sun, J. (2022) · ISA Transactions · doi:10.1016/j.isatra.2022.02.034 · Cited by 7
Liu, Y., Qiu, G., Gou, W., & Liu, Q. (2023) · Optimal Control Applications and Methods · doi:10.1002/oca.2977 · Cited by 7
Zhang, H., & Yan, Z. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127269 · Cited by 7
Lin, S.-W., Chu, C., & Tung, C.-F. (2024) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2024.3351800 · Cited by 7
“One class of model-free algorithms, called reinforcement learning (RL), also known as approximate or adaptive dynamic programming (ADP), can also be applied for MG autonomous operations [11], [12], [13], [14], [15], [16], [17], [18].”
BackgroundXu, H., & Carrillo, L. R. G. (2017) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2262877 · Cited by 7
“Next, substituting the (19) into (18), the relationship between HJB residual error and critic NN weight estimation errors for UAV_i can be derived as”
BackgroundDynamic Routing Algorithm with Q-learning for Internet of things with Delayed Estimator
Fang, W., Feng, R., & Chen, H. (2019) · IOP Conference Series Earth and Environmental Science · doi:10.1088/1755-1315/234/1/012048 · Cited by 7
“In practical problems, the agent should interact with the environment so as to achieve its goal [14].”
BackgroundUnderstanding the mechanism of human–computer game: a distributed reinforcement learning perspective
Peng, Z., Hu, J., Zhao, Y., & Ghosh, B. K. (2020) · International Journal of Systems Science · doi:10.1080/00207721.2020.1803436 · Cited by 7
Assured learning‐enabled autonomy: A metacognitive reinforcement learning framework
Mustafa, A., Mazouchi, M., Nageshrao, S. P., & Modares, H. (2021) · International Journal of Adaptive Control and Signal Processing · doi:10.1002/acs.3326 · Cited by 7
“In these traditional RL-based controllers, the reinforcement signal feedback is derived through a fixed quadratic objective function [23]-[24].”
BackgroundPolicy set iteration for Markov decision processes
Chang, H. S. (2013) · Automatica · doi:10.1016/j.automatica.2013.09.010 · Cited by 7
Goharimanesh, M., Jannatabadi, E. A., Moeinkhah, H., Sistani, M. B. N., & Akbari, A. A. (2017) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-161211 · Cited by 7
Recurrent neural network-based fault detector for aileron failures of aircraft
Yoshikawa, N., Belkhir, N., & Suzuki, S. (2017) · Asian Control Conference (ASCC) · doi:10.1109/ascc.2017.8287391 · Cited by 7
“Regarding such topic, reinforcement learning, as been widely proved in the literature (see [22], [23], [24], [25] for an overview of reinforcement for feedback control and adaptive controllers) to be a promising path to explore.”
BackgroundFlight controller learning based on real-time model estimation of a quadrotor aircraft
Alabsi, M. I., & Fields, T. D. (2018) · Proceedings of the Institution of Mechanical Engineers Part G Journal of Aerospace Engineering · doi:10.1177/0954410018795524 · Cited by 7
Ma, B., Li, R., Dong, B., Yu, M., Zhou, F., & Li, Y. (2018) · Chinese Automation Congress · doi:10.1109/cac.2018.8623365 · Cited by 7
“There are many unique outstanding features of ADP for working out the optimal control problem of nonlinear systems [6].”
BackgroundAI Intelligence for the Grid 16 Years Later: Progress, Challenges and Lessons for Other Sectors
Werbos, P. J. (2018) · 2018 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489560 · Cited by 6
“Exciting results have been reported by several teams (e.g. [11]) showing how RLADP can substantially improve performance in such applications.”
BackgroundInfluentialFang, Y., Li, Y., & Tong, S. (2024) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2024.3510589 · Cited by 6
Inverse Reinforcement Learning for Disturbed Networked Nonlinear Systems With Data Dropouts
Shi, P., Xue, W., Fan, J., Lewis, F. L., & Lian, B. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3626762 · Cited by 6
“By applying RL and adaptive dynamic programming (ADP) [15], the works [16], [17], [18] address H ∞ control problems leveraging online measurement data, eliminating the need for full knowledge of system dynamics.”
Control Allocation of Spacecraft Revisited: A Stackelberg Differential Game Approach
Ni, Y., Yang, H., & Jiang, B. (2024) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2024.3386154 · Cited by 6
“An approximate dynamic programming methodology offers a promising approach to approximate these solutions effectively, even when subjected to the amplitude constraint of the control law [18], [19].”
Backgroundda Silva, F. N., & Neto, J. V. D. F. (2021) · IEEE Access · doi:10.1109/access.2021.3080626 · Cited by 6
Expanding human visual field: online learning of assistive camera views by an aerial co-robot
Bentz, W., Qian, L., & Panagou, D. (2022) · Autonomous Robots · doi:10.1007/s10514-022-10059-4 · Cited by 6
“The robot’s action policy is trained online using Q-learning (Lewis and Liu 2013).”
MethodInfluentialPoznyak, A. S., Noriega-Marquez, S., Hernández-Sánchez, A., Ballesteros, M., & Chaírez, I. (2023) · Mathematics · doi:10.3390/math11051211 · Cited by 6
Amirparast, A., & Sani, S. K. H. (2023) · International Journal of Systems Science · doi:10.1080/00207721.2023.2221240 · Cited by 6
“The RL algorithms have been widely used in both offline and online frameworks to obtain optimal state feedback gains in LQT and LQR (Bertsekas, 2019; Kamalapurkar et al., 2018; Lewis & Liu, 2013; Lewis & Vrabie, 2009; Vamvoudakis et al., 2021).”
MethodSolving optimal predictor-feedback control using approximate dynamic programming
Wang, H., Zhao, F., Zhang, Z., Xu, J., & Li, X. (2024) · Automatica · doi:10.1016/j.automatica.2024.111848 · Cited by 6
Talaei, B., Xu, H., & Jagannathan, S. (2014) · The proceedings of the IEEE Conference on Control Applications/The proceedings of the ... IEEE Conference on Control Applications · doi:10.1109/isic.2014.6967634 · Cited by 6
“dominant finite set of states can be estimated and the key inputs inside the system domain can be controlled by taking advantage of state of the art adaptive dynamic programming (ADP) schemes for finite dimensional systems [9].”
BackgroundTrinh, H. T., Bae, S.-H., & Tran, Q. D. (2023) · PROMET - Traffic&Transportation · doi:10.7307/ptt.v35i5.246 · Cited by 8
“In recent years, reinforcement learning (RL) has been studied and applied in several fields, such as robots or game management [5].”
BackgroundHuang, J., Xu, D., Li, Y., & Ma, Y. (2024) · Mathematics · doi:10.3390/math12081146 · Cited by 6
Silva, M. E. G., Neto, J. V. D. F., & de Souza, F. D. C. (2014) · UKSim International Conference on Computer Modelling and Simulation (UKSim) · doi:10.1109/uksim.2014.108 · Cited by 6
“feedback procedure in time [7].”
Data-driven robust optimal control design for uncertain cascaded systems using value iteration
Bian, T., & Jiang, Z. (2015) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2015.7403422 · Cited by 6
Overview of Adaptive Dynamic Programming
Liu, D., Wei, Q., Wang, D., Yang, X., & Li, H. (2017) · Advances in industrial control · doi:10.1007/978-3-319-50815-3_1 · Cited by 6
Adaptive H∞ Tracking Control of Nonlinear Systems Using Reinforcement Learning
Modares, H., Kiumarsi, B., Vamvoudakis, K. G., & Lewis, F. L. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-812976-0.00018-x · Cited by 6
Metodología de programación dinámica aproximada para control óptimo basada en datos
Díaz, H., Armesto, L., & Sala, A. (2019) · Revista Iberoamericana de Automática e Informática Industrial RIAI · doi:10.4995/riai.2019.10379 · Cited by 6
A Novel Model-Free Actor-Critic Reinforcement Learning Approach for Dynamic Target Tracking
Elhussein, A., & Miah, S. (2020) · doi:10.1109/mic50194.2020.9209618 · Cited by 6
“The objective of the optimization problem, following [19], is to find an optimal sequence of control polices { u ∗ k } ∞ k =0 that minimizes the cost index J along the state-trajectories (2) and (1).”
BackgroundPolicy search for active fault diagnosis with partially observable state
Král, L., & Punĉochář, I. (2022) · International Journal of Adaptive Control and Signal Processing · doi:10.1002/acs.3456 · Cited by 6
“Then, the criterion (17) is approximated as J ( γ ) ≈ E {R (h)} , (20) where R ∶ → R+ is a cumulative cost function defined as”
BackgroundFernandez‐Gauna, B., Osa, J. L., & Graña, M. (2014) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-07995-0_13 · Cited by 6
Bahrami, V., Masouleh, M. T., & Kalhor, A. (2024) · Mechanism and Machine Theory · doi:10.1016/j.mechmachtheory.2024.105676 · Cited by 5
Sun, Z., & Jia, G. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129758 · Cited by 5
“In recent years, significant progress has been made in solving optimal control problems with unknown models using reinforcement learning (RL) [1, 2] and approximate / adaptive dynamic programming (ADP) [3, 4].”
Approximate optimal adaptive control for weakly coupled nonlinear systems: A neuro‐inspired approach
Carrillo, L. R. G., Vamvoudakis, K. G., & Hespanha, J. P. (2015) · International Journal of Adaptive Control and Signal Processing · doi:10.1002/acs.2631 · Cited by 5
Stability Analysis for Incremental Adaptive Dynamic Programming with Approximation Errors
Li, Y., & Kampen, E.-J. V. (2023) · Journal of Aerospace Engineering · doi:10.1061/jaeeez.aseng-5097 · Cited by 5
Hao, L., Wang, C., & Shi, Y. (2024) · Mathematics · doi:10.3390/math12101533 · Cited by 5
Policy gradient methods for designing dynamic output feedback controllers
Sadamoto, T., & Hirai, T. (2024) · European Journal of Control · doi:10.1016/j.ejcon.2024.101081 · Cited by 5
Li, H., Liu, D., Wang, D., & Li, C. (2014) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2014.6889467 · Cited by 5
“Adaptive dynamic programming (ADP) [3]–[5], also known as approximate dynamic programming [6]–[8] or neuro-dynamic programming [9], has received significantly increasing attention, which has been applied in many practical areas, such as call admission control [10], engine control [11], and energy system control [12],…”
BackgroundPuncochaf, I., Škach, J., & Ŝimandl, M. (2015) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2015.09.576 · Cited by 5
Efficient Actor-Critic Algorithm with Hierarchical Model Learning and Planning
Zhong, S., Liu, Q., & Fu, Q. (2016) · Computational Intelligence and Neuroscience · doi:10.1155/2016/4824072 · Cited by 5
“To avoid the curse of dimensionality in DP, approximate dynamic programming (ADP) received much attention to obtain approximate solutions of the Hamilton-JacobiBellman (HJB) equation by combining DP, RL, and function approximation [8].”
MethodSecurity Strategies and Multi-Criteria Decision Making
Raß, S. (2018) · Static & dynamic game theory: foundations & applications · doi:10.1007/978-3-319-75268-6_3 · Cited by 5
Machine Learning for Agile and Self-Adaptive Congestion Management in Active Distribution Networks
Babar, M., Roos, M. H., & Nguyen, P. H. (2019) · IEEE International Conference on Environment and Electrical Engineering and IEEE Industrial and Commercial Power Systems Europe (EEEIC / I&CPS Europe) · doi:10.1109/eeeic.2019.8783624 · Cited by 6
Feng, Y., Wang, C., & Li, A. (2020) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2020.06.113 · Cited by 5
Miah, S. (2017) · IEEE International Symposium on Robotics and Intelligent Sensors (IRIS) · doi:10.1109/iris.2017.8250096 · Cited by 5
“Since the approximate dynamic programming method is a part of conventional optimal control techniques [25], the problem boils down to find the sequence of control inputs {uk}k=0 that minimizes the infinite-horizon cost function defined by”
MethodCyberphysical Security Methods
Mahmoud, M. S., & Xia, Y. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-816119-7.00017-4 · Cited by 5
Greene, M. L., Deptula, P., Nivison, S., & Dixon, W. E. (2019) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc40024.2019.9029500 · Cited by 5
“…function approximators, such as neural networks (NNs), can be used to approximate the value function by using the state as the training inputs; however, it is difficult to ensure sufficient learning using traditional approaches since the system needs to be sufficiently exciting (cf., [22]–[24]).”
BackgroundAdaptive optimal control for linear stochastic systems with additive noise
Bian, T., & Zhong-Ping, J. (2015) · Chinese Control Conference (CCC) · doi:10.1109/chicc.2015.7260102 · Cited by 5
“See also [20–22] for three review papers, and [23, 24] for two recent books.”
BackgroundLATERAL CONTROL FOR AUTONOMOUS LAND VEHICLES VIA DUAL HEURISTIC PROGRAMMING
Huang, Z., Lian, C., Xu, X., & Wang, J. (2016) · International Journal of Robotics and Automation · doi:10.2316/journal.206.2016.6.206-4878 · Cited by 5
Adaptive Variants of Optimal Feedback PoliciesPreprint
Lopez, B. T., & Slotine, J.-J. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.02709 · Cited by 5
Decentralized Event-Sampled Control of Multi-Unit Power Systems via Adaptive Dynamic Programming
Yang, X., Meng, J., Zhang, S., & Ge, L. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3649059 · Cited by 4
“Nonetheless, analytical solutions of HJBEs are usually unavailable [6].”
Symbiotic Control of Uncertain Dynamical Systems*
Yucelen, T., Sarsılmaz, S. B., & Yildirim, E. (2024) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc56724.2024.10885962 · Cited by 4
“For example, robust control [1,2] and sliding mode control [3,4] are well-known fixed-gain control approaches, whereas adaptive control [5,6,7] and reinforcement learning [8,9] are well-known adaptive learning approaches.”
Lyapunov-based adaptive deep system identification for approximate dynamic programming
Makumi, W. A., Patil, O. S., & Dixon, W. E. (2025) · Automatica · doi:10.1016/j.automatica.2025.112462 · Cited by 4
Robust adaptive dynamic programming for continuous-time linear stochastic systems
Bian, T., & Jiang, Z. (2014) · The proceedings of the IEEE Conference on Control Applications/The proceedings of the ... IEEE Conference on Control Applications · doi:10.1109/isic.2014.6967601 · Cited by 4
“See also [26], [27] for two review papers, and [28] for a recent book.”
BackgroundValue Iteration ADP for Discrete-Time Nonlinear Systems
Liu, D., Wei, Q., Wang, D., Yang, X., & Li, H. (2017) · Advances in industrial control · doi:10.1007/978-3-319-50815-3_2 · Cited by 4
Zhao, F., Jiang, Z., & Liu, T. (2019) · Chinese Control Conference (CCC) · doi:10.23919/chicc.2019.8866079 · Cited by 5
“Adaptive dynamic programming (ADP) combines the ideas of adaptive control, optimal control, and reinforcement learning, and has been proved effective in dealing with adaptive optimal control problems [1–5].”
BackgroundStochastic Two-Player Zero-Sum Learning Differential Games
Liu, M., Wan, Y., Lewis, F. L., & Lopez, V. G. (2019) · International Conference on Control and Automation · doi:10.1109/icca.2019.8899568 · Cited by 4
“Reinforcement learning (RL), a subarea of machine learning, was developed based on the idea that successful decisions should be remembered as a reinforcement signal, such that they are more likely to be used in future decisions [7].”
BackgroundAn Optimal Solution to Infinite Horizon Nonlinear Control Problems
Mohamed, M. N. G., Goyal, R., & Chakravorty, S. (2023) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc49753.2023.10384307 · Cited by 4
“Approximate dynamic programming methods [6, 7] typically use aggregation methods [8] [9] or various function approximation methods [10] [11] to give an approximately optimal policy/value function with a high confidence.”
MethodDifferential neural network robust constrained controller using approximate dynamic programming
Noriega-Marquez, S., Poznyak, A. S., Hernández-Sánchez, A., & Chaírez, I. (2024) · European Journal of Control · doi:10.1016/j.ejcon.2024.101003 · Cited by 4
Adaptive Finite-Time-Based Neural Optimal Control of Time-Delayed Wheeled Mobile Robotics Systems
Li, S., Ren, T., Ding, L., & Liu, L. (2024) · Sensors · doi:10.3390/s24175462 · Cited by 4
“…the importance of optimal control, many scholars have conducted extensive research on optimal control algorithms and have obtained certain achievements, mainly including two optimization methods [10,11], adaptive dynamic programming (ADP) methods [12] and reinforcement learning (RL) methods [13].”
BackgroundOptimal Digital Control with Uncertain Network Delay of Linear Systems Using Reinforcement Learning
Fujita, T., & Ushio, T. (2016) · IEICE Transactions on Fundamentals of Electronics Communications and Computer Sciences · doi:10.1587/transfun.e99.a.454 · Cited by 4
Data-driven robust regulation of nonlinear systems with mismatched disturbances
Yang, X., & He, H. (2017) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci.2017.8285268 · Cited by 4
Data-Driven Adaptive Optimal Tracking Control for Completely Unknown Systems
Hou, D., Na, J., Gao, G., & Li, G. (2018) · IEEE Data Driven Control and Learning Systems Conference (DDCLS) · doi:10.1109/ddcls.2018.8515964 · Cited by 4
“Recently, by incorporating adaptive control into the dynamic programming (DP) [4], a new methodology, named adaptive dynamic programming (ADP), has been proposed to solve optimal control problem [1, 5-11], where neural networks (NNs) are used to approximate the optimal cost function and then derive the optimal control…”
MethodClosed-loop Model Selection for Kernel-based Models Using Bayesian Optimization
Beckers, T. L. (2019) · mediaTUM (Technical University of Munich) · Cited by 4
A linear programming methodology for approximate dynamic programming
Díaz, H., Sala, A., & Armesto, L. (2020) · International Journal of Applied Mathematics and Computer Science · doi:10.34768/amcs-2020-0028 · Cited by 4
Poveda, J. I., & Teel, A. R. (2021) · Studies in systems, decision and control · doi:10.1007/978-3-030-60990-0_24 · Cited by 4
LQR Approach to Aircraft Control Based on the Adaptive Critic Design
Chulin, M. I., Tiumentsev, Y. V., & Zarubin, R. A. (2023) · Studies in computational intelligence · doi:10.1007/978-3-031-44865-2_44 · Cited by 4
Degradation Tolerant Control Learning for Discrete-Time Affine Nonlinear Systems
Kanso, S., Jha, M., & Theilliol, D. (2023) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2023.10.1178 · Cited by 4
Adversarial Attacks to Direct Data-Driven Control for Destabilization
Sasahara, H. (2023) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc49753.2023.10383531 · Cited by 4
Computational intelligence from AI to BI to NI
Werbos, P. J. (2015) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2191520 · Cited by 4
“He was the person who first understood and explained the modern concept of cardinal utility function [7], essential to a rigorous approach to reinforcement learning [2,3] and to modern practices of risk management and decision trees [8].”
BackgroundA boundedness theoretical analysis for GrADP design: A case study on maze navigation
Ni, Z., Zhong, X., & He, H. (2015) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280475 · Cited by 4
Fujita, T., & Ushio, T. (2014) · IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning · doi:10.1109/adprl.2014.7010632 · Cited by 4
“Recently, it has been applied to the design of optimal controllers for systems with unknown dynamics [11], [13], [14], [18].”
BackgroundAdaptive Feature Pursuit: Online Adaptation of Features in Reinforcement Learning
Bhatnagar, S., Borkar, V. S., & Prashanth, L. A. (2012) · Wiley eBooks · doi:10.1002/9781118453988.ch23 · Cited by 4
Learning-based Event-triggered Adaptive Optimal Output Regulation of Linear Discrete-time Systems
Zhao, F., Gao, W., Liu, T., & Jiang, Z. (2021) · IEEE Data Driven Control and Learning Systems Conference (DDCLS) · doi:10.1109/ddcls52934.2021.9455453 · Cited by 4
Memristors for More Than Just Memory: How to Use Learning to Expand Applications
Werbos, P. J. (2012) · Springer eBooks · doi:10.1007/978-94-007-4491-2_5 · Cited by 4
Reinforcement Learning for Robotic Time-optimal Path Tracking Using Prior KnowledgePreprint
Xiao, J., Li, L., Zou, Y., & Zhang, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.00388 · Cited by 4
Direct heuristic dynamic programming design with extreme learning machine
Luo, X., Lv, Y., Wang, W., Zhou, M., & Zhao, W. (2016) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727440 · Cited by 3
“I. INTRODUCTION The adaptive dynamic programming (ADP) was proposed to obtain approximate solutions to the Hamilton-Jacobi-Bellman (HJB) equation and has achieved good performance in optimal control problems [1], [2].”
MethodAmirparast, A., & Sani, S. K. H. (2024) · International Journal of Systems Science · doi:10.1080/00207721.2024.2364289 · Cited by 3
“…reinforcement learning (RL) algorithms and adaptive dynamic programming have emerged as effective approaches to address this challenge for both discrete and continuous time systems (Bertsekas, 2019; Kamalapurkar et al., 2018; Lewis & Liu, 2013; Lewis & Vrabie, 2009; Vamvoudakis et al., 2021).”
Reinforcement learning: Application to PWR's core power level control
Kirgni, H. B. (2025) · Progress in Nuclear Energy · doi:10.1016/j.pnucene.2025.105644 · Cited by 6
Robust policy iteration for the continuous-time stochastic H ∞ control problem with unknown dynamics
Sun, Z., & Jia, G. (2025) · Mathematics and Computers in Simulation · doi:10.1016/j.matcom.2025.09.009 · Cited by 3
An Actor-Critic Framework for Online Control With Environment Stability Guarantee
Osinenko, P., Yaremenko, G., Malaniya, G., & Bolychev, A. (2023) · IEEE Access · doi:10.1109/access.2023.3306070 · Cited by 3
Metadata
- Authors
- Fairbank, M., Prokhorov, D., & Alonso, E.
- Affiliations
- The University of Texas at Arlington (US)
- University of Illinois Chicago (US)
- Venue
- Wiley eBooks
- Publisher
- Wiley
- Published
- 2012-12-17
- OpenAlex
- W560518094
- Cited by
- 577
- References
- 33
- Citations by year
- 2026: 15 · 2025: 27 · 2024: 38 · 2023: 48 · 2022: 38 · 2021: 50 · 2020: 44 · 2019: 64 · 2018: 35 · 2017: 54 · 2016: 43 · 2015: 39 · 2014: 48 · 2013: 27 · 2012: 7
- FWCI
- 6.42
- Type
- book
- Language
- en
- Open access
- closed
- Licence
- doi.wiley.com/10.1002/tdm_license_1.1 (tdm)
- Locations
- Wiley eBooks (publishedVersion)
- City Research Online (City University London) (acceptedVersion)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Advanced Control Systems Optimization
- Keywords
- reinforcement learning, approximate dynamic programming, feedback control, adaptive dynamic programming, multiplayer games, decision-making, robot control, industrial process control, economic decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.