Reinforcement Learning and Approximate Dynamic Programming for Feedback Control
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
In this chapter, we extend the ADP algorithm, dual heuristic programming (DHP), to include a “bootstrapping” parameter λ, analogous to that used in the reinforcement learning algorithm TD(λ). The resulting algorithm, which we call VGL(λ) for value-gradient learning, is proven to produce a weight update that can be equivalent to backpropagation through time (BPTT) applied to a greedy policy on a critic function. This provides a surprising connection between the two alternate methods of BPTT and DHP. Under certain smoothness conditions, VGL(λ=1) with a greedy policy acquires the strong convergence conditions of BPTT, while using a general function approximator for the critic. We show that this can lead to increased stability in the learning of control problems by a neural network
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“ADP represents a kind of data-based control technique which can guarantee the stability of the feedback control system [9].”
From Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach (2023) · cites this paper as [9]
In the citing paper
“Even if an approximated model of the HSA can be developed, the dynamic uncertainty, produced by the mismatch between the approximated model and the true HSA model, will degrade the control performance of the traditionally designed optimal controller [9,12].”
From Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach (2023) · cites this paper as [9]
In the citing paper
“Adaptive dynamic programming (ADP) ensures an e ective way to achieve high performance of the optimal controller which relies on adaptive control, optimal control and reinforcement learning [9, 23-26].”
From Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach (2023) · cites this paper as [9]
Citations
Cites
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem
Vamvoudakis, K. G., & Lewis, F. L. (2010) · Automatica · doi:10.1016/j.automatica.2010.02.018 · Cited by 1,703
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Cited by 1,427
An analysis of temporal-difference learning with function approximation
Tsitsiklis, J. N., & Van Roy, B. (1997) · IEEE Transactions on Automatic Control · doi:10.1109/9.580874 · Cited by 1,194
Prokhorov, D. V., & Wunsch, D. C. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.623201 · Cited by 1,186
Self-citationDiscrete-Time Nonlinear HJB Solution Using Approximate Dynamic Programming: Convergence Proof
Al-Tamimi, A., Lewis, F. L., & Abu-Khalaf, M. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.926614 · Cited by 1,075
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Medical Entomology and Zoology · Cited by 971
Reinforcement Learning in Continuous Time and Space
Doya, K. (2000) · Neural Computation · doi:10.1162/089976600300015961 · Cited by 960
Adaptive Dynamic Programming: An Introduction
Wang, F.-Y., Zhang, H., & Liu, D. (2009) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2009.932261 · Cited by 898
Applied Optimal Control and Estimation
Lewis, F. L. (1992) · Prentice Hall PTR eBooks · Cited by 431
Adaptive-critic-based neural networks for aircraft optimal control
Balakrishnan, S. N., & Biega, V. (1996) · Journal of Guidance, Control, and Dynamics · doi:10.2514/3.21715 · Cited by 261
Padhi, R., Unnikrishnan, N., Wang, X., & Balakrishnan, S. N. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.08.010 · Cited by 216
Online Adaptive Critic Flight Control
Ferrari, S., & Stengel, R. F. (2004) · Journal of Guidance Control and Dynamics · doi:10.2514/1.12597 · Cited by 163
Dual heuristic programming excitation neurocontrol for generators in a multimachine power system
Venayagamoorthy, G. K., Harley, R. G., & Wunsch, D. C. (2003) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2003.809438 · Cited by 130
Continuous-Time Adaptive Critics
Hanselmann, T., Noakes, L., & Zaknich, A. (2007) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2006.889499 · Cited by 124
Issues on Stability of ADP Feedback Controllers for Dynamical Systems
Balakrishnan, S. N., Ding, J., & Lewis, F. L. (2008) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2008.926599 · Cited by 108
Solving Optimal Continuous Thrust Rendezvous Problems with Generating Functions
Park, C., Guibout, V., & Scheeres, D. J. (2006) · Journal of Guidance Control and Dynamics · doi:10.2514/1.14580 · Cited by 101
State-constrained agile missile control with adaptive-critic-based neural networks
Han, D., & Balakrishnan, S. N. (2002) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2002.1014669 · Cited by 99
Using ADP to Understand and Replicate Brain Intelligence: The Next Level Design?
Werbos, P. J. (2007) · Springer eBooks · doi:10.1007/978-3-540-73267-9_6 · Cited by 78
Padhi, R., & Balakrishnan, S. N. (2003) · Neural Networks · doi:10.1016/s0893-6080(03)00131-x · Cited by 62
Adaptive critic design for intelligent steering and speed control of a 2-axle vehicle
Lendaris, G. G., Schultz, L., & Shannon, T. T. (2000) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2000.861283 · Cited by 42
The divergence of reinforcement learning algorithms with value-iteration and function approximation
Fairbank, M., & Alonso, E. (2012) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2012.6252792 · Cited by 33
Self-citationReinforcement Learning by Value GradientsPreprint
Fairbank, M. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0803.3539 · Cited by 24
Self-citationLi, B., & Si, J. (2007) · IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning · doi:10.1109/adprl.2007.368175 · Cited by 23
Finite-horizon input-constrained nonlinear optimal control using single network adaptive critics
Heydari, A., & Balakrishnan, S. N. (2011) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.2011.5991378 · Cited by 19
Padhi, R., & Balakrishnan, S. N. (2004) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc.2004.1386805 · Cited by 12
Fairbank, M., & Alonso, E. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1101.0428 · Cited by 12
Self-citationAdaptive critic neural network force controller for atomic force microscope-based nanomanipulation
Yang, Q., & Jagannathan, S. (2006) · The proceedings of the IEEE Conference on Control Applications/The proceedings of the ... IEEE Conference on Control Applications · doi:10.1109/cacsd-cca-isic.2006.4776690 · Cited by 10
Ding, J., Balakrishnan, S. N., & Lewis, F. L. (2010) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2010.5596588 · Cited by 9
Robust/Optimal Temperature Profile Control Using Neural Networks
Yadav, V. H., Padhi, R., & Balakrishnan, S. N. (2006) · The proceedings of the IEEE Conference on Control Applications/The proceedings of the ... IEEE Conference on Control Applications · doi:10.1109/cacsd-cca-isic.2006.4777145 · Cited by 8
Cited by
Adaptive Dynamic Programming for Control: A Survey and Recent Advances
Liu, D., Xue, S., Zhao, B., Luo, B., & Wei, Q. (2021) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2020.3042876 · Cited by 708
Modares, H., & Lewis, F. L. (2014) · Automatica · doi:10.1016/j.automatica.2014.05.011 · Cited by 701
“Strong connections between RL and optimal control have prompted a major effort towards introducing and developing online and model-free RL algorithms to learn the solution to optimal control problems (Lewis & Liu, 2012; Vrabie, Vamvoudakis, & Lewis, 2013; Zhang, Liu, Luo, & Wang, 2012).”
BackgroundKiumarsi, B., Lewis, F. L., Modares, H., Karimpour, A., & Naghibi-Sistani, M.-B. (2014) · Automatica · doi:10.1016/j.automatica.2014.02.015 · Cited by 574
Liu, D., Yang, X., Wang, D., & Wei, Q. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2417170 · Cited by 440
“By utilizing the gradient descent algorithm, the weight tuning law for the critic NN is generally given as [31]–[34], [40], and [41]”
BackgroundMethodInfluentialAdaptive Dynamic Programming and Adaptive Optimal Output Regulation of Linear Systems
Gao, W., & Jiang, Z.-P. (2016) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2016.2548662 · Cited by 413
“, [10], [12], [19]–[21], [23], [24], [26], [30], [31], [33], [35].”
Metadata
- Authors
- Fairbank, M., Prokhorov, D., & Alonso, E.
- Affiliations
- The University of Texas at Arlington (US)
- University of Illinois Chicago (US)
- Venue
- Wiley eBooks
- Publisher
- Wiley
- Published
- 2012-12-17
- OpenAlex
- W560518094
- Cited by
- 577
- References
- 33
- Citations by year
- 2026: 15 · 2025: 27 · 2024: 38 · 2023: 48 · 2022: 38 · 2021: 50 · 2020: 44 · 2019: 64 · 2018: 35 · 2017: 54 · 2016: 43 · 2015: 39 · 2014: 48 · 2013: 27 · 2012: 7
- FWCI
- 6.42
- Type
- book
- Language
- en
- Open access
- closed
- Licence
- doi.wiley.com/10.1002/tdm_license_1.1 (tdm)
- Locations
- Wiley eBooks (publishedVersion)
- City Research Online (City University London) (acceptedVersion)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Adaptive Dynamic Programming Control
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Advanced Control Systems Optimization
- Keywords
- reinforcement learning, approximate dynamic programming, feedback control, adaptive dynamic programming, multiplayer games, decision-making, robot control, industrial process control, economic decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.