Reinforcement Learning for Partially Observable Dynamic Processes: Adaptive Dynamic Programming Using Measured Output Data
Abstract
Approximate dynamic programming (ADP) is a class of reinforcement learning methods that have shown their importance in a variety of applications, including feedback control of dynamical systems. ADP generally requires full information about the system internal states, which is usually not available in practical situations. In this paper, we show how to implement ADP methods using only measured input/output data from the system. Linear dynamical systems with deterministic behavior are considered herein, which are systems of great interest in the control system community. In control system theory, these types of methods are referred to as output feedback (OPFB). The stochastic equivalent of the systems dealt with in this paper is a class of partially observable Markov decision processes. We develop both policy iteration and value iteration algorithms that converge to an optimal controller that requires only OPFB. It is shown that, similar to Q -learning, the new methods have the important advantage that knowledge of the system dynamics is not needed for the implementation of these learning algorithms or for the OPFB control. Only the order of the system, as well as an upper bound on its "observability index," must be known. The learned OPFB controller is in the form of a polynomial autoregressive moving-average controller that has equivalent performance with the optimal state variable feedback gain.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Optimal feedback control systems have been applied to basic sensorimotor control and to reinforcement learning models.
“Optimal feedback control systems have been applied to basic sensorimotor control [97] and to reinforcement learning models [98].”
From Health behavior models in the age of mobile interventions: are our theories up to the task? (2011) · cites this paper as [98]
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,406 · see citing works
(2012) · Springer eBooks · doi:10.1007/978-1-4419-1428-6_5086 · Cited by 3,601 · see citing works
Lewis, F. L. (1986) · Cited by 2,957 · see citing works
Self-citationBeyond regression : new fools for prediction and analysis in the behavioral sciences
Werbos, P. J. (1974) · Medical Entomology and Zoology · Cited by 2,746 · see citing works
Approximate Dynamic Programming
Powell, W. B. (2011) · Wiley series in probability and statistics · doi:10.1002/9781118029176 · Cited by 2,359 · see citing works
Kronecker products and matrix calculus in system theory
Brewer, J. (1978) · IEEE Transactions on Circuits and Systems · doi:10.1109/tcs.1978.1084534 · Cited by 1,920 · see citing works
Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem
Vamvoudakis, K. G., & Lewis, F. L. (2010) · Automatica · doi:10.1016/j.automatica.2010.02.018 · Cited by 1,722 · see citing works
Self-citationApproximate Dynamic Programming
Powell, W. B. (2007) · Wiley series in probability and statistics · doi:10.1002/9780470182963 · Cited by 1,582 · see citing works
Reinforcement learning and adaptive dynamic programming for feedback control
Lewis, F. L., & Vrabie, D. (2009) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2009.933854 · Cited by 1,569 · see citing works
Self-citationLancaster, P., & Rodman, L. (1995) · doi:10.1093/oso/9780198537953.001.0001 · Cited by 1,218 · see citing works
Prokhorov, D. V., & Wunsch, D. C. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.623201 · Cited by 1,181 · see citing works
Discrete-Time Nonlinear HJB Solution Using Approximate Dynamic Programming: Convergence Proof
Al-Tamimi, A., Lewis, F. L., & Abu-Khalaf, M. (2008) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2008.926614 · Cited by 1,073 · see citing works
Self-citationIterative feedback tuning: theory and applications
Hjalmarsson, H., GEVERS, M. R., Gunnarsson, S., & Lequin, O. (1998) · IEEE Control Systems · doi:10.1109/37.710876 · Cited by 995 · see citing works
(2010) · Springer eBooks · doi:10.1007/978-0-387-30164-8_588 · Cited by 978 · see citing works
Ioannou, P., & Fi̇dan, B. (2006) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9780898718652 · Cited by 977 · see citing works
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Medical Entomology and Zoology · Cited by 971 · see citing works
Reinforcement Learning in Continuous Time and Space
Doya, K. (2000) · Neural Computation · doi:10.1162/089976600300015961 · Cited by 961 · see citing works
Adaptive Dynamic Programming: An Introduction
Wang, F., Zhang, H., & Liu, D. (2009) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2009.932261 · Cited by 901 · see citing works
Handbook of Learning and Approximate Dynamic Programming
Si, J., Barto, A. G., Powell, W. B., & Wunsch, D. C. (2004) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544785 · Cited by 863 · see citing works
Stabilization of Nonlinear Uncertain Systems
Krstić, M., Modestino, J. W., Deng, H., Fettweis, A., Massey, J. L., Thoma, M., Sontag, E. D., & Dickinson, B. (1998) · Cited by 626 · see citing works
The unfalsified control concept and learning
Safonov, M. G., & Tsao, T. (1997) · IEEE Transactions on Automatic Control · doi:10.1109/9.587340 · Cited by 597 · see citing works
Widrow, B. C., & Bilello, M. (2002) · IEEE International Symposium on Intelligent Control · doi:10.1109/isic.1993.397732 · Cited by 593 · see citing works
Schultz, W. (2004) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2004.03.017 · Cited by 517 · see citing works
Adaptive linear quadratic control using policy iteration
Bradtke, S. J., Ydstie, B. E., & Barto, A. G. (2005) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.1994.735224 · Cited by 416 · see citing works
Rogers, S. C. (1997) · Control Engineering Practice · doi:10.1016/s0967-0661(97)89242-8 · Cited by 328 · see citing works
Neural networks for control and system identification
Werbos, P. J. (2003) · Proceedings of the IEEE Conference on Decision & Control, including the Symposium on Adaptive Processes · doi:10.1109/cdc.1989.70114 · Cited by 326 · see citing works
Hewer, G. (1971) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1971.1099755 · Cited by 326 · see citing works
Stochastic Learning and Optimization
Cao, X. (2007) · Springer eBooks · doi:10.1007/978-0-387-69082-7 · Cited by 243 · see citing works
Model-free control of nonlinear stochastic systems with discrete-time measurements
Spall, J. C., & Cristion, J. A. (1998) · IEEE Transactions on Automatic Control · doi:10.1109/9.718605 · Cited by 241 · see citing works
Lewis, F. L., Xie, L., & Popa, D. O. (2017) · CRC Press eBooks · doi:10.1201/9781315221656 · Cited by 222 · see citing works
Self-citationVirtual reference feedback tuning for two degree of freedom controllers
Lecchini, A., Campi, M. C., & Savaresi, S. M. (2002) · International Journal of Adaptive Control and Signal Processing · doi:10.1002/acs.711 · Cited by 158 · see citing works
Q-learning and Pontryagin's Minimum Principle
Mehta, P. G., & Meyn, S. (2009) · doi:10.1109/cdc.2009.5399753 · Cited by 122 · see citing works
Optimal design of adaptive tracking controllers for non-linear systems
Li, Z., & Krstić, M. (1997) · Automatica · doi:10.1016/s0005-1098(97)00072-1 · Cited by 110 · see citing works
Neural mechanisms of learning and control
Doya, K., Kimura, H., & Kawato, M. (2001) · IEEE Control Systems · doi:10.1109/37.939943 · Cited by 103 · see citing works
Aangenent, W., Kostic, D., de Jager, B., van de Molengraft, R., & Steinbuch, M. (2005) · Proceedings of the 2005, American Control Conference, 2005. · doi:10.1109/acc.2005.1470171 · Cited by 70 · see citing works
Adaptive optimal controllers based on Generalized Policy Iteration in a continuous-time framework
Vrabie, D., Vamvoudakis, K. G., & Lewis, F. L. (2009) · Mediterranean Conference on Control and Automation (MED) · doi:10.1109/med.2009.5164743 · Cited by 62 · see citing works
Self-citationOnline actor critic algorithm to solve the continuous-time infinite horizon optimal control problem
Vamvoudakis, K. G., & Lewis, F. L. (2009) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2009.5178586 · Cited by 57 · see citing works
Self-citationLewis, F. L., Liu, D., & Lendaris, G. G. (2008) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2008.925890 · Cited by 53 · see citing works
Self-citationMarkov Data-Based LQG Control1
Shi, G., & Skelton, R. E. (1998) · Journal of Dynamic Systems Measurement and Control · doi:10.1115/1.1286868 · Cited by 49 · see citing works
Vamvoudakis, K. G., Vrabie, D., & Lewis, F. L. (2009) · IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning · doi:10.1109/adprl.2009.4927523 · Cited by 46 · see citing works
Self-citationOptimal design of adaptive tracking controllers for nonlinear systems
Li, Z., & Krstić, M. (1997) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.1109/acc.1997.609721 · Cited by 36 · see citing works
State-Space System Identification with Identified Hankel Matrix
Lim, R. K., Phan, M. Q., & Longman, R. W. (1998) · Cited by 28 · see citing works
Unifying Input-Output and State-Space Perspectives of Predictive Control
Phan, M. Q., Lim, R. K., & Longman, R. W. (1998) · Cited by 26 · see citing works
Identification of a Multistep-Ahead Observer and Its Application to Predictive Control
Lim, R. K., & Phan, M. Q. (1997) · Journal of Guidance Control and Dynamics · doi:10.2514/2.4177 · Cited by 20 · see citing works
Suppressing non-periodically repeating disturbances in mechanical servo systems
Tousain, R. L., Boissy, J.-C., Norg, M. L., Steinbuch, M., & Bosgra, O. H. (2002) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.1998.757832 · Cited by 15 · see citing works
Minimum time pulse response based control of flexible structures
Bennighof, J. K., Chang, S.-H. M., & Subramaniam, M. (1993) · Journal of Guidance Control and Dynamics · doi:10.2514/3.21095 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Lewis, F. L., Vrabie, D., & Vamvoudakis, K. G. (2012) · IEEE Control Systems · doi:10.1109/mcs.2012.2214134 · Cited by 1,107 · see citing works
Self-citationJiang, Y., & Jiang, Z. (2012) · Automatica · doi:10.1016/j.automatica.2012.06.096 · Cited by 1,079 · see citing works
Health behavior models in the age of mobile interventions: are our theories up to the task?
Riley, W. T., Rivera, D. E., Atienza, A. A., Nilsen, W., Allison, S. M., & Mermelstein, R. (2011) · Translational Behavioral Medicine · doi:10.1007/s13142-011-0021-7 · Cited by 978 · see citing works
Optimal and Autonomous Control Using Reinforcement Learning: A Survey
Kiumarsi, B., Vamvoudakis, K. G., Modares, H., & Lewis, F. L. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2773458 · Cited by 908 · see citing works
Self-citationZhang, H., Cui, L., Zhang, X., & Luo, Y. (2011) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2011.2168538 · Cited by 645 · see citing works
Kiumarsi, B., Lewis, F. L., Modares, H., Karimpour, A., & Sistani, M. B. N. (2014) · Automatica · doi:10.1016/j.automatica.2014.02.015 · Cited by 575 · see citing works
Self-citationZhang, H., Cui, L., & Luo, Y. (2012) · IEEE Transactions on Cybernetics · doi:10.1109/tsmcb.2012.2203336 · Cited by 465 · see citing works
Modares, H., & Lewis, F. L. (2014) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2014.2317301 · Cited by 451 · see citing works
Self-citationLiu, D., Yang, X., Wang, D., & Wei, Q. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2417170 · Cited by 439 · see citing works
Robust Adaptive Dynamic Programming and Feedback Stabilization of Nonlinear Systems
Jiang, Y., & Jiang, Z. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2294968 · Cited by 415 · see citing works
Value iteration and adaptive dynamic programming for data-driven adaptive optimal control design
Bian, T., & Jiang, Z.-P. (2016) · Automatica · doi:10.1016/j.automatica.2016.05.003 · Cited by 348 · see citing works
Model-Free Optimal Tracking Control via Critic-Only Q-Learning
Luo, B., Liu, D., Huang, T., & Wang, D. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2585520 · Cited by 332 · see citing works
Finite-Approximation-Error-Based Optimal Control Approach for Discrete-Time Nonlinear Systems
Liu, D., & Wei, Q. (2012) · IEEE Transactions on Cybernetics · doi:10.1109/tsmcb.2012.2216523 · Cited by 304 · see citing works
Liu, D., Li, H., & Wang, D. (2014) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2013.2295351 · Cited by 263 · see citing works
An Event-Triggered ADP Control Approach for Continuous-Time System With Unknown Internal States
Zhong, X., & He, H. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2523878 · Cited by 263 · see citing works
Optimal Tracking Control of Unknown Discrete-Time Linear Systems Using Input-Output Measured Data
Kiumarsi, B., Lewis, F. L., Sistani, M. B. N., & Karimpour, A. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2384016 · Cited by 251 · see citing works
Self-citationOff-Policy Actor-Critic Structure for Optimal Control of Unknown Systems With Disturbances
Song, R., Lewis, F. L., Wei, Q., & Zhang, H. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2421338 · Cited by 241 · see citing works
Self-citationVamvoudakis, K. G. (2016) · Systems & Control Letters · doi:10.1016/j.sysconle.2016.12.003 · Cited by 240 · see citing works
Self-citationReinforcement learning and optimal adaptive control: An overview and implementation examples
Khan, S. G., Herrmann, G., Lewis, F. L., Pipe, T., & Melhuish, C. (2012) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2012.03.004 · Cited by 235 · see citing works
Self-citationGao, W., Jiang, Y., Jiang, Z.-P., & Chai, T. (2016) · Automatica · doi:10.1016/j.automatica.2016.05.008 · Cited by 234 · see citing works
Song, R., Lewis, F. L., & Wei, Q. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582849 · Cited by 234 · see citing works
Self-citationLiu, L., Wang, Z., & Zhang, H. (2016) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2016.2517155 · Cited by 232 · see citing works
Policy Gradient Adaptive Dynamic Programming for Data-Based Optimal Control
Luo, B., Liu, D., Wu, H., Wang, D., & Lewis, F. L. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2623859 · Cited by 224 · see citing works
Self-citationZhang, K., Su, R., Zhang, H., & Tian, Y. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3053269 · Cited by 215 · see citing works
Distributed Optimal Consensus Control for Nonlinear Multiagent System With Unknown Dynamic
Zhang, J., Zhang, H., & Feng, T. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2728622 · Cited by 205 · see citing works
Leader-to-Formation Stability of Multiagent Systems: An Adaptive Optimal Control Approach
Gao, W., Jiang, Z., Lewis, F. L., & Wang, Y. (2018) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2018.2799526 · Cited by 194 · see citing works
Self-citationModel-Freeλ-Policy Iteration for Discrete-Time Linear Quadratic Regulation
Yang, Y., Kiumarsi, B., Modares, H., & Xu, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3098985 · Cited by 194 · see citing works
Lee, J. Y., Park, J. B., & Choi, Y. H. (2012) · Automatica · doi:10.1016/j.automatica.2012.06.008 · Cited by 191 · see citing works
Finite-Approximation-Error-Based Discrete-Time Iterative Adaptive Dynamic Programming
Wei, Q., Wang, F., Liu, D., & Yang, X. (2014) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2354377 · Cited by 188 · see citing works
Modares, H., Lewis, F. L., & Jiang, Z. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2477810 · Cited by 181 · see citing works
Self-citationQin, C., Qiao, X., Wang, J., Zhang, D., Hou, Y., & Hu, S. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3302656 · Cited by 174 · see citing works
Improved Deep Hybrid Networks for Urban Traffic Flow Prediction Using Trajectory Data
Duan, Z., Yang, Y., Zhang, K., Ni, Y., & Bajgain, S. (2018) · IEEE Access · doi:10.1109/access.2018.2845863 · Cited by 162 · see citing works
Jiang, Y., & Jiang, Z. (2012) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2012.2213353 · Cited by 155 · see citing works
Zhong, X., He, H., Zhang, H., & Wang, Z. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2305841 · Cited by 153 · see citing works
Adaptive $Q$ -Learning for Data-Based Optimal Output Regulation With Experience Replay
Luo, B., Yang, Y., & Liu, D. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2821369 · Cited by 142 · see citing works
Multiple Actor-Critic Structures for Continuous-Time Optimal Control Using Input-Output Data
Song, R., Lewis, F. L., Wei, Q., Zhang, H., Jiang, Z., & Levine, D. S. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2399020 · Cited by 140 · see citing works
Self-citationRizvi, S. A. A., & Lin, Z. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2886735 · Cited by 139 · see citing works
Relations between Model Predictive Control and Reinforcement Learning
Görges, D. (2017) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2017.08.747 · Cited by 136 · see citing works
A Learning-Based Power Management Method for Networked Microgrids Under Incomplete Information
Zhang, Q., Dehghanpour, K., Wang, Z., & Huang, Q. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2933502 · Cited by 136 · see citing works
Luo, B., Liu, D., & Wu, H. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2751018 · Cited by 134 · see citing works
Fault-tolerant control of a hydraulic servo actuator via adaptive dynamic programming
Stojanović, V. (2023) · Mathematical Modelling and Control · doi:10.3934/mmc.2023016 · Cited by 136 · see citing works
Robust Adaptive Dynamic Programming
Jiang, Y., & Jiang, Z. (2017) · Wiley eBooks · doi:10.1002/9781119132677.ch5 · Cited by 138 · see citing works
Optimal control for discrete-time affine non-linear systems using general value iteration
Li, H., & Liu, D. (2012) · IET Control Theory and Applications · doi:10.1049/iet-cta.2011.0783 · Cited by 133 · see citing works
Rizvi, S. A. A., & Lin, Z. (2018) · Automatica · doi:10.1016/j.automatica.2018.05.027 · Cited by 129 · see citing works
Data-Based Optimal Control of Multiagent Systems: A Reinforcement Learning Design Approach
Zhang, J., Wang, Z., & Zhang, H. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2868715 · Cited by 125 · see citing works
Data-Driven $H_\infty$ Control for Nonlinear Distributed Parameter Systems
Luo, B., Huang, T., Wu, H., & Yang, X. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2461023 · Cited by 120 · see citing works
Output Feedback Q-Learning Control for the Discrete-Time Linear Quadratic Regulator Problem
Rizvi, S. A. A., & Lin, Z. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2870075 · Cited by 124 · see citing works
Zhang, H., Zhang, K., Xiao, G., & Jiang, H. (2019) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2018.2889377 · Cited by 119 · see citing works
Ding, L., Li, S., Gao, H., Liu, Y., Huang, L., & Deng, Z. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2939424 · Cited by 113 · see citing works
Adaptive Suboptimal Output-Feedback Control for Linear Systems Using Integral Reinforcement Learning
Zhu, L. M., Modares, H., Gan, O. P., Lewis, F. L., & Yue, B. (2014) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2014.2322778 · Cited by 108 · see citing works
Self-citationDjordjevic, V., Tao, H., Song, X., He, S., Gao, W., & Stojanovic, V. (2023) · Mathematical Biosciences and Engineering · doi:10.3934/mbe.2023376 · Cited by 107 · see citing works
“Motivated by [44,47], the discrete-time HSA described by (3.”
MethodRobust Adaptive Dynamic Programming With an Application to Power Systems
Jiang, Y., & Jiang, Z. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2249668 · Cited by 106 · see citing works
Liu, D., Yang, X., & Li, H. (2012) · Neural Computing and Applications · doi:10.1007/s00521-012-1249-y · Cited by 102 · see citing works
Model‐based vs data‐driven adaptive control: An overview
Benosman, M. (2018) · International Journal of Adaptive Control and Signal Processing · doi:10.1002/acs.2862 · Cited by 101 · see citing works
Continuous-Time Q-Learning for Infinite-Horizon Discounted Cost Linear Quadratic Regulator Problems
Muthukumar, P., Modares, H., Lewis, F. L., & Aurangzeb, M. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2322116 · Cited by 97 · see citing works
Self-citationData-driven control of hydraulic servo actuator based on adaptive dynamic programming
Djordjevic, V., Stojanović, V., Tao, H., Song, X., He, S., & Gao, W. (2022) · Discrete and Continuous Dynamical Systems - S · doi:10.3934/dcdss.2021145 · Cited by 93 · see citing works
Mu, C., Wang, D., & He, H. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2752845 · Cited by 91 · see citing works
Adaptive Dynamic Programming for Stochastic Systems With State and Control Dependent Noise
Bian, T., Jiang, Y., & Jiang, Z. (2016) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2016.2550518 · Cited by 90 · see citing works
Chen, C., Xie, L., Xie, K., Lewis, F. L., & Xie, S. (2022) · Automatica · doi:10.1016/j.automatica.2022.110581 · Cited by 85 · see citing works
Self-citationReinforcement Q-Learning Algorithm for H ∞ Tracking Control of Unknown Discrete-Time Linear Systems
Peng, Y., Chen, Q., & Sun, W. (2019) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2019.2957000 · Cited by 84 · see citing works
Luo, B., & Wu, H. (2012) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2012.2194781 · Cited by 82 · see citing works
Xu, N., Kong, Y., Chu, L., Ju, H., Zhi-hua, Y., Xu, Z., & Xu, Z. (2019) · Applied Sciences · doi:10.3390/app9102026 · Cited by 82 · see citing works
Wei, Q., & Liu, D. (2014) · Neurocomputing · doi:10.1016/j.neucom.2013.09.069 · Cited by 79 · see citing works
Chen, C., Xie, L., Jiang, Y., Xie, K., & Xie, S. (2022) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3172590 · Cited by 79 · see citing works
Song, R., & Lewis, F. L. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.01.082 · Cited by 78 · see citing works
Self-citationRădac, M., Precup, R., & Roman, R. (2018) · ISA Transactions · doi:10.1016/j.isatra.2018.01.014 · Cited by 76 · see citing works
Self-Learning Cruise Control Using Kernel-Based Least Squares Policy Iteration
Wang, J., Xu, X., Liu, D., Sun, Z., & Chen, Q. (2013) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2013.2271276 · Cited by 76 · see citing works
Li, S., Ding, L., Gao, H., Liu, Y., Huang, L., & Deng, Z. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2900326 · Cited by 76 · see citing works
Data-driven model-free slip control of anti-lock braking systems using reinforcement Q-learning
Rădac, M., & Precup, R. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.08.036 · Cited by 75 · see citing works
Adaptive Optimal Output Regulation of Time-Delay Systems via Measurement Feedback
Gao, W., & Jiang, Z. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2850520 · Cited by 75 · see citing works
Li, M., Wen, Y., Gao, X. H., Si, J., & Huang, H. (2021) · IEEE Transactions on Robotics · doi:10.1109/tro.2021.3078317 · Cited by 76 · see citing works
Tutsoy, Ö., Barkana, D. E., & Tugal, H. (2018) · ISA Transactions · doi:10.1016/j.isatra.2018.03.002 · Cited by 73 · see citing works
H∞ Tracking Control of Discrete-Time System With Delays via Data-Based Adaptive Dynamic Programming
Liu, Y., Zhang, H., Yu, R., & Xing, Z. (2019) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2019.2946397 · Cited by 70 · see citing works
Linear Quadratic Control Using Model-Free Reinforcement Learning
Yaghmaie, F. A., Gustafsson, F., & Lennart Ljung, A. (2022) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3145632 · Cited by 71 · see citing works
Data-Driven Coordinated Attack Policy Design Based on Adaptive $\mathcal {L}_2$ -Gain Optimal Theory
An, L., & Yang, G. (2017) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2017.2753466 · Cited by 70 · see citing works
H ∞ Tracking Control for Linear Discrete-Time Systems: Model-Free Q-Learning Designs
Yang, Y., Wan, Y., Zhu, J., & Lewis, F. L. (2020) · IEEE Control Systems Letters · doi:10.1109/lcsys.2020.3001241 · Cited by 65 · see citing works
Self-citationNoel, M. M., & Pandian, B. J. (2014) · Applied Soft Computing · doi:10.1016/j.asoc.2014.06.037 · Cited by 64 · see citing works
Data-Based Adaptive Dynamic Programming for a Class of Discrete-Time Systems With Multiple Delays
Zhang, H., Liu, Y., Xiao, G., & Jiang, H. (2017) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2017.2758849 · Cited by 64 · see citing works
Data-Driven Distributed Output Consensus Control for Partially Observable Multiagent Systems
Jiang, H., & He, H. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2788819 · Cited by 64 · see citing works
Zhang, X., Zhang, H., Sun, Q., & Luo, Y. (2012) · Neurocomputing · doi:10.1016/j.neucom.2012.01.025 · Cited by 62 · see citing works
Zargarzadeh, H., Dierks, T., & Jagannathan, S. (2013) · International Journal of Adaptive Control and Signal Processing · doi:10.1002/acs.2432 · Cited by 61 · see citing works
Yang, X., Liu, D., Wang, D., & Wei, Q. (2014) · Neural Networks · doi:10.1016/j.neunet.2014.03.008 · Cited by 61 · see citing works
Jiang, Y., Liu, L., & Feng, G. (2024) · Automatica · doi:10.1016/j.automatica.2024.111531 · Cited by 60 · see citing works
An adaptive decision-making method with fuzzy Bayesian reinforcement learning for robot soccer
Shi, H., Lin, Z., Zhang, S., Li, X., & Hwang, K. (2018) · Information Sciences · doi:10.1016/j.ins.2018.01.032 · Cited by 60 · see citing works
Zhao, B., & Li, Y. (2018) · International Journal of Control Automation and Systems · doi:10.1007/s12555-016-0711-5 · Cited by 59 · see citing works
LQ Secure Control for Cyber-Physical Systems Against Sparse Sensor and Actuator Attacks
An, L., & Yang, G. (2018) · IEEE Transactions on Control of Network Systems · doi:10.1109/tcns.2018.2878507 · Cited by 59 · see citing works
Rădac, M., Precup, R., & Roman, R. (2016) · International Journal of Systems Science · doi:10.1080/00207721.2016.1236423 · Cited by 58 · see citing works
Peng, Z., Zhao, Y., Hu, J., Luo, R., Ghosh, B. K., & Nguang, S. K. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2021.3050768 · Cited by 58 · see citing works
Zhou, Y., Kampen, E.-J. V., & Chu, Q. P. (2016) · Journal of Guidance Control and Dynamics · doi:10.2514/1.g001762 · Cited by 58 · see citing works
Off-Policy Reinforcement Learning for Tracking in Continuous-Time Systems on Two Time Scales
Xue, W., Fan, J., Lopez, V. G., Jiang, Y., Chai, T., & Lewis, F. L. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3017461 · Cited by 58 · see citing works
Self-citationYan, Y., Zhang, H., Sun, J., & Wang, Y. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3241070 · Cited by 57 · see citing works
Zhang, J., Zhang, H., Luo, Y., & Feng, T. (2014) · Neurocomputing · doi:10.1016/j.neucom.2013.12.038 · Cited by 57 · see citing works
Reduced-dimensional reinforcement learning control using singular perturbation approximations
Mukherjee, S., Bai, H., & Chakrabortty, A. (2021) · Automatica · doi:10.1016/j.automatica.2020.109451 · Cited by 57 · see citing works
Opacity Enforcement for Confidential Robust Control in Linear Cyber-Physical Systems
An, L., & Yang, G. (2019) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2019.2925498 · Cited by 55 · see citing works
Output feedback adaptive dynamic programming for linear differential zero-sum games
Rizvi, S. A. A., & Lin, Z. (2020) · Automatica · doi:10.1016/j.automatica.2020.109272 · Cited by 54 · see citing works
Mu, C., & Wang, K. (2018) · Nonlinear Dynamics · doi:10.1007/s11071-018-4713-0 · Cited by 53 · see citing works
Optimal control of a two‐wheeled self‐balancing robot by reinforcement learning
Guo, L., Rizvi, S. A. A., & Lin, Z. (2020) · International Journal of Robust and Nonlinear Control · doi:10.1002/rnc.5058 · Cited by 53 · see citing works
Ding, S. X., & Li, L. (2021) · Control Engineering Practice · doi:10.1016/j.conengprac.2021.104790 · Cited by 52 · see citing works
Cui, X., Zhang, H., Luo, Y., & Pei-fu, Z. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.12.021 · Cited by 52 · see citing works
Output-feedback Robust Tracking Control of Uncertain Systems via Adaptive Learning
Zhao, J., & Lv, Y. (2023) · International Journal of Control Automation and Systems · doi:10.1007/s12555-021-0882-6 · Cited by 52 · see citing works
- Authors
- Lewis, F. L., & Vamvoudakis, K. G.
- Affiliations
- The University of Texas at Arlington (US)
- Venue
- IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)
- Published
- 2011-02
- Volume
- 41
- Issue
- 1
- Pages
- 14-25
- ISSN
- 1083-4419 (ISSN-L)
- 1941-0492
- OpenAlex
- W2151966330
- Cited by
- 544
- References
- 48
- Citations by year
- 2026: 22 · 2025: 49 · 2024: 46 · 2023: 43 · 2022: 37 · 2021: 49 · 2020: 53 · 2019: 34 · 2018: 52 · 2017: 26 · 2016: 35 · 2015: 19 · 2014: 33 · 2013: 18 · 2012: 19
- FWCI
- 15.12
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Advanced Control Systems Optimization
- Keywords
- approximate dynamic programming, partially observable Markov decision processes, output feedback control, linear dynamical systems, policy iteration, value iteration, Q-learning, observability index
- MeSH terms
- Algorithms, Artificial Intelligence, Feedback, Learning, Markov Chains, Reinforcement, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
