The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.
Adaptive Dynamic Programming: An Introduction
Abstract
In this article, we introduce some recent research trends within the field of adaptive/approximate dynamic programming (ADP), including the variations on the structure of ADP schemes, the development of ADP algorithms and applications of ADP schemes. For ADP algorithms, the point of focus is that iterative algorithms of ADP can be sorted into two classes: one class is the iterative algorithm with initial stable policy; the other is the one without the requirement of initial stable policy. It is generally believed that the latter one has less computation at the cost of missing the guarantee of system stability during iteration process. In addition, many recent papers have provided convergence analysis associated with the algorithms developed. Furthermore, we point out some topics for future studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
(2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_5086 · Cited by 3,515
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,233
Lewis, F. L. (1986) · Cited by 2,948
Approximate Dynamic Programming: Solving the Curses of Dimensionality
Powell, W. B. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9781118029176 · Cited by 2,297
A New Approach to Manipulator Control: The Cerebellar Model Articulation Controller (CMAC)
Albus, J. S. (1975) · Journal of Dynamic Systems, Measurement, and Control · doi:10.1115/1.3426922 · Cited by 2,177 · cited 1×
“In 1991, Lin and Kim [51] integrate the cerebellar model articulation controller technique [1] with the box-based scheme.”
MethodApproximate Dynamic Programming: Solving the Curses of Dimensionality
Powell, W. B. (2007) · Wiley Series in Probability and Statistics · doi:10.1002/9780470182963 · Cited by 1,566
Abu-Khalaf, M., & Lewis, F. L. (2005) · Automatica · doi:10.1016/j.automatica.2004.11.034 · Cited by 1,349
On an iterative technique for Riccati equation computations
Kleinman, D. (1968) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1968.1098829 · Cited by 1,210
Prokhorov, D. V., & Wunsch, D. C. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.623201 · Cited by 1,186
Discrete-Time Nonlinear HJB Solution Using Approximate Dynamic Programming: Convergence Proof
Al-Tamimi, A., Lewis, F. L., & Abu-Khalaf, M. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.926614 · Cited by 1,075
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Medical Entomology and Zoology · Cited by 971
(2011) · Encyclopedia of Machine Learning · doi:10.1007/978-0-387-30164-8_588 · Cited by 950
(1991) · The MIT Press eBooks · doi:10.7551/mitpress/4939.001.0001 · Cited by 873
Handbook of Learning and Approximate Dynamic Programming
Si, J., Barto, A. G., Powell, W. B., & Wunsch, D. (2004) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544785 · Cited by 856
Practical issues in temporal difference learning
Tesauro, G. (1992) · Machine Learning · doi:10.1007/bf00992697 · Cited by 787
Online learning control by association and reinforcement
Si, J., & Wang, Y.-T. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.914523 · Cited by 773
Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches
White, D. A., & Sofge, D. (1992) · Medical Entomology and Zoology · Cited by 705
Murray, J. J., Cox, C. J., Lendaris, G. G., & Saeks, R. (2002) · IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2002.801727 · Cited by 677 · cited 5×
“[2], [3], [5], [8], [11]–[13], [21], [22], [25], [30], [31], [34], [35], [40], [46], [49], [52], [54], [55], [63], [70], [76], [80], [83], [95], [96], [99], [100].”
MethodInfluentialGalerkin approximations of the generalized Hamilton-Jacobi-Bellman equation
Beard, R. W., Saridis, G. N., & Wen, J. T.-Y. (1997) · Automatica · doi:10.1016/s0005-1098(97)00128-3 · Cited by 667
Zhang, H., Luo, Y., & Liu, D. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2009.2027233 · Cited by 649
Self-citationAl-Tamimi, A., Lewis, F. L., & Abu-Khalaf, M. (2007) · Automatica · doi:10.1016/j.automatica.2006.09.019 · Cited by 611 · cited 1×
“In addition, practical applications of ADP with significant economic impact are of great demand.”
BackgroundA Menu of Designs for Reinforcement Learning Over Time
Werbos, P. J. (1991) · Neural Networks for Control · doi:10.7551/mitpress/4939.003.0007 · Cited by 605
Zhang, H., Wei, Q., & Luo, Y. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.920269 · Cited by 489
Self-citationThe Art and Theory of Dynamic Programming
(1977) · Mathematics in Science and Engineering · doi:10.1016/s0076-5392(08)x6068-0 · Cited by 474
Powell, W. B. (2007) · Cited by 471
Applied Optimal Control and Estimation
Lewis, F. L. (1992) · Prentice Hall PTR eBooks · Cited by 431
Adaptive linear quadratic control using policy iteration
Bradtke, S. J., Ydstie, B. E., & Barto, A. G. (2005) · Proceedings of 1994 American Control Conference - ACC '94 · doi:10.1109/acc.1994.735224 · Cited by 404
An Approximation Theory of Optimal Control for Trainable Manipulators
Saridis, G. N., & Lee, C.-S. G. (1979) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1979.4310171 · Cited by 348
Punish/Reward: Learning with a Critic in Adaptive Threshold Systems
Widrow, B., Gupta, N. K., & Maitra, S. (1973) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1973.4309272 · Cited by 280
Adaptive-critic-based neural networks for aircraft optimal control
Balakrishnan, S. N., & Biega, V. (1996) · Journal of Guidance, Control, and Dynamics · doi:10.2514/3.21715 · Cited by 261
Werbos, P. (1987) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1987.289329 · Cited by 251
Helicopter trimming and tracking control using direct neural dynamic programming
Enns, R., & Si, J. (2003) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2003.813839 · Cited by 243
The art and theory of dynamic programming
Rota, G.-C. (1978) · Advances in Mathematics · doi:10.1016/0001-8708(78)90080-4 · Cited by 243
He, P., & Jagannathan, S. (2007) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2006.883869 · Cited by 235
Abu-Khalaf, M., Lewis, F. L., & Huang, J. (2006) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2006.884959 · Cited by 229
Venayagamoorthy, G. K., Harley, R. G., & Wunsch, D. C. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2002.1000146 · Cited by 220
Padhi, R., Unnikrishnan, N., Wang, X., & Balakrishnan, S. N. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.08.010 · Cited by 216
Adaptive critic designs: A case study for neurocontrol
Prokhorov, D. V., Santiago, R. A., & Wunsch, D. C. (1995) · Neural Networks · doi:10.1016/0893-6080(95)00042-9 · Cited by 199
Consistency of HDP applied to a simple reinforcement learning problem
Werbos, P. J. (1990) · Neural Networks · doi:10.1016/0893-6080(90)90088-3 · Cited by 194
Approximate Solutions to the Time-Invariant Hamilton–Jacobi–Bellman Equation
Beard, R. W., Saridis, G. N., & Wen, J. T.-Y. (1998) · Journal of Optimization Theory and Applications · doi:10.1023/a:1022664528457 · Cited by 192
Ensemble Algorithms in Reinforcement Learning
Wiering, M. A., & van Hasselt, H. (2008) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.920231 · Cited by 185
Adaptive Critic Designs for Discrete-Time Zero-Sum Games With Application to $H_{\infty}$ Control
Al-Tamimi, A., Abu-Khalaf, M., & Lewis, F. L. (2007) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2006.880135 · Cited by 184
Online Adaptive Critic Flight Control
Ferrari, S., & Stengel, R. F. (2004) · Journal of Guidance Control and Dynamics · doi:10.2514/1.12597 · Cited by 163
A Self-Learning Call Admission Control Scheme for CDMA Cellular Networks
Liu, D., Zhang, Y., & Zhang, H. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2005.853408 · Cited by 162
Self-citationAdaptive Critic Learning Techniques for Engine Torque and Air–Fuel Ratio Control
Liu, D., Javaherian, H., Kovalenko, O., & Huang, T. (2008) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2008.922019 · Cited by 154
Self-citationThe Art and Theory of Dynamic Programming
Unwin, A. R., Dreyfus, S. E., & Law, A. M. (1979) · Journal of the Operational Research Society · doi:10.2307/3009654 · Cited by 142
Action-dependent adaptive critic designs
Liu, D., Xiaoxu Xiong, A., & Zhang, Y. (2002) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2001.939495 · Cited by 134
Self-citationReinforcement Learning-Based Output Feedback Control of Nonlinear Systems With Input Constraints
He, P., & Jagannathan, S. (2005) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2004.840124 · Cited by 124
Continuous-Time Adaptive Critics
Hanselmann, T., Noakes, L., & Zaknich, A. (2007) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2006.889499 · Cited by 124
Reinforcement Learning and Distributed Local Model Synthesis
Landelius, T. (1997) · Cited by 113
Issues on Stability of ADP Feedback Controllers for Dynamical Systems
Balakrishnan, S. N., Ding, J., & Lewis, F. L. (2008) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2008.926599 · Cited by 108
Missile defense and interceptor allocation by neuro-dynamic programming
Bertsekas, D. P., Homer, M. L., Logan, D. A., Patek, S. D., & SANDELL, N. R. (2000) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/3468.823480 · Cited by 106
Wei, Q., Zhang, H., & Dai, J. (2008) · Neurocomputing · doi:10.1016/j.neucom.2008.05.012 · Cited by 101
Self-citationCMAC-based adaptive critic self-learning control
Lin, C.-S., & Kim, H. (1991) · IEEE Transactions on Neural Networks · doi:10.1109/72.134290 · Cited by 94
Hamilton–Jacobi–Bellman Equations and Approximate Dynamic Programming on Time Scales
Seiffertt, J., Sanyal, S., & Wunsch, D. C. (2008) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2008.923532 · Cited by 92
Park, J.-W., Harley, R. G., & Venayagamoorthy, G. K. (2003) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2003.816493 · Cited by 86
Training strategies for critic and action neural networks in dual heuristic programming method
Lendaris, G. G., & Paintz, C. (2002) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1997.616109 · Cited by 82
Convergence analysis of adaptive critic based optimal control
Liu, X., & Balakrishnan, S. N. (2000) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.2000.879538 · Cited by 80
Using ADP to Understand and Replicate Brain Intelligence: The Next Level Design?
Werbos, P. J. (2007) · Springer eBooks · doi:10.1007/978-3-540-73267-9_6 · Cited by 78
Mohagheghi, S., Venayagamoorthy, G. K., & Harley, R. G. (2006) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2006.882467 · Cited by 68
Robust/Optimal Temperature Profile Control of a High-Speed Aerospace Vehicle Using Neural Networks
Yadav, V. H., Padhi, R., & Balakrishnan, S. N. (2007) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.899229 · Cited by 66
Discrete-time nonlinear HJB solution using Approximate dynamic programming: Convergence Proof
Al-Tamimi, A., & Lewis, F. L. (2007) · IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning · doi:10.1109/adprl.2007.368167 · Cited by 61
Apache Helicopter Stabilization Using Neural Dynamic Programming
Enns, R., & Si, J. (2002) · Journal of Guidance Control and Dynamics · doi:10.2514/2.4870 · Cited by 59
Yen, G. G., & DeLima, P. G. (2005) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2005.844122 · Cited by 50
Continuous-Time ADP for Linear Systems with Partially Unknown Dynamics
Vrabie, D., Abu-Khalaf, M., Lewis, F. L., & Wang, Y. (2007) · IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning · doi:10.1109/adprl.2007.368195 · Cited by 48
A neighboring optimal adaptive critic for missile guidance
Dalton, J. S., & Balakrishnan, S. N. (1996) · Mathematical and Computer Modelling · doi:10.1016/0895-7177(95)00226-x · Cited by 36
Reinforcement learning algorithms for robotic navigation in dynamic environments
Yen, G. G., & Hickey, T. (2004) · ISA Transactions · doi:10.1016/s0019-0578(07)60032-9 · Cited by 31
Discrete-Time Adaptive Dynamic Programming using Wavelet Basis Function Neural Networks
Jin, N., Liu, D., Huang, T., & Pang, Z. (2007) · IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning · doi:10.1109/adprl.2007.368180 · Cited by 31
Self-citationHigher Level Application of ADP: A Next Phase for the Control Field?
Lendaris, G. G. (2008) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2008.918073 · Cited by 28
Second-Order Training of Adaptive Critics for Online Process Control
Govindhasamy, J. J., McLoone, S., & Irwin, G. W. (2005) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2004.843276 · Cited by 25
Dynamic re-optimization of a fed-batch fermentor using adaptive critic designs
Iyer, M. S., & Wunsch, D. C. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.963778 · Cited by 23
Li, B., & Si, J. (2007) · IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning · doi:10.1109/adprl.2007.368175 · Cited by 23
Padhi, R., & Balakrishnan, S. N. (2006) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2006.876633 · Cited by 23
The Adaptive Dynamic Programming Theorem
Murray, J., Cox, C. J., & Saeks, R. E. (2003) · Birkhäuser Boston eBooks · doi:10.1007/978-1-4612-0037-6_19 · Cited by 16
Shih, P., Kaul, B. C., Jagannathan, S., & Drallmeier, J. A. (2007) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/ijcnn.2007.4370989 · Cited by 11
Model-free Approximate Dynamic Programming Schemes for Linear Systems
Al-Tamimi, A., Vrabie, D., Abu-Khalaf, M., & Lewis, F. L. (2007) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/ijcnn.2007.4370985 · Cited by 11
Dankert, J. R., Yang, L., & Si, J. (2006) · The proceedings of the IEEE Conference on Control Applications/The proceedings of the ... IEEE Conference on Control Applications · doi:10.1109/cacsd-cca-isic.2006.4776689 · Cited by 7
Cited by
Data-Driven Intelligent Transportation Systems: A Survey
Zhang, J., Wang, F.-Y., Wang, K., Lin, W.-H., Xu, X., & Chen, C. (2011) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2011.2158001 · Cited by 1,817
“One promising way is to develop adaptive dynamic programming (ADP) and reinforcement learning (RL) methods [88] for the performance optimization of complex dynamic systems, because ADP and RL have been shown to be powerful in solving Markov decision processes with large or continuous state and action spaces.”
BackgroundSelf-citationReinforcement learning and adaptive dynamic programming for feedback control
Lewis, F. L., & Vrabie, D. (2009) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2009.933854 · Cited by 1,558
(2012) · IEEE Control Systems · doi:10.1109/mcs.2012.2214134 · Cited by 1,105
Jiang, Y., & Jiang, Z.-P. (2012) · Automatica · doi:10.1016/j.automatica.2012.06.096 · Cited by 1,072
Adaptive Dynamic Programming for Control: A Survey and Recent Advances
Liu, D., Xue, S., Zhao, B., Luo, B., & Wei, Q. (2021) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2020.3042876 · Cited by 708
Self-citation
Metadata
- Authors
- Wang, F.-Y., Zhang, H., & Liu, D.
- Venue
- IEEE Computational Intelligence Magazine
- Published
- 2009-05
- Volume
- 4
- Issue
- 2
- Pages
- 39-47
- ISSN
- 1556-603X (ISSN-L)
- OpenAlex
- W2050838777
- Cited by
- 898
- References
- 82
- FWCI
- 36.70
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.