The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Safe Reinforcement Learning Using Robust MPC
Open accessAbstract
Reinforcement learning (RL) has recently impressed the world with stunning results in various applications. While the potential of RL is now well established, many critical aspects still need to be tackled, including safety and stability issues. These issues, while secondary for the RL community, are central to the control community that has been widely investigating them. Model predictive control (MPC) is one of the most successful control techniques because, among others, of its ability to provide such guarantees even for uncertain constrained systems. Since MPC is an optimization-based technique, optimality has also often been claimed. Unfortunately, the performance of MPC is highly dependent on the accuracy of the model used for predictions. In this article, we propose to combine RL and MPC in order to exploit the advantages of both, and therefore, obtain a controller that is optimal and safe. We illustrate the results with two numerical examples in simulations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
(2006) · Springer Series in Operations Research and Financial Engineering · doi:10.1007/978-0-387-40065-5 · Cited by 9,008
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Practical methods of optimization
(1988) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(88)90129-2 · Cited by 5,818
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
Model predictive control: Theory and practice—A survey
García, C. E., Prett, D. M., & Morari, M. (1989) · Automatica · doi:10.1016/0005-1098(89)90002-2 · Cited by 5,284
(1999) · Cited by 5,147
Deterministic Policy Gradient Algorithms
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014) · HAL · Cited by 3,275
Model predictive control: Recent developments and future promise
Mayne, D. Q. (2014) · Automatica · doi:10.1016/j.automatica.2014.10.128 · Cited by 1,975
Reinforcement learning and adaptive dynamic programming for feedback control
Lewis, F. L., & Vrabie, D. (2009) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2009.933854 · Cited by 1,558
(2015) · Journal of Machine Learning Research · Cited by 1,119
Numerical Optimization (Springer Series in Operations Research and Financial Engineering)
Nocedal, J., & Wright, S. J. (2000) · Cited by 1,112
(2012) · IEEE Control Systems · doi:10.1109/mcs.2012.2214134 · Cited by 1,105
(2013) · Dynamic Modelling and Predictive Control in Solid Oxide Fuel Cells: First Principle and Data‐Based Approaches · doi:10.1002/9781118501054.ch6 · Cited by 1,047
Practical methods of optimization; (2nd ed.)
Fletcher, R. (1987) · Wiley-Interscience eBooks · Cited by 964
Recursive state estimation for a set-membership description of uncertainty
Bertsekas, D., & Rhodes, I. (1971) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1971.1099674 · Cited by 719
Systems with persistent disturbances: predictive control with restricted constraints
Chisci, L., Rossiter, J. A., & Zappa, G. (2001) · Automatica · doi:10.1016/s0005-1098(01)00051-6 · Cited by 676
Provably safe and robust learning-based model predictive control
Aswani, A., Gonzalez, H., Sastry, S. S., & Tomlin, C. (2013) · Automatica · doi:10.1016/j.automatica.2013.02.003 · Cited by 539
An Application of Reinforcement Learning to Aerobatic Helicopter Flight
Abbeel, P., Coates, A., Quigley, M., & Ng, A. Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0006 · Cited by 535
A Lyapunov Function for Economic Optimizing Model Predictive Control
Diehl, M., Amrit, R., & Rawlings, J. B. (2010) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2010.2101291 · Cited by 513
Economic optimization using model predictive control with a terminal cost
Amrit, R., Rawlings, J. B., & Angeli, D. (2011) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2011.10.011 · Cited by 399
Economic receding horizon control without terminal constraints
Grüne, L. (2013) · Automatica · doi:10.1016/j.automatica.2012.12.003 · Cited by 363
Safe Model-based Reinforcement Learning with Stability Guarantees
Berkenkamp, F., Turchetta, M., Schoellig, A. P., & Krause, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08551 · Cited by 299
Data-Driven Economic NMPC Using Reinforcement Learning
Gros, S., & Zanon, M. (2020) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2019.2913768 · Cited by 269
Self-citationSafe Exploration in Continuous Action Spaces
Dalal, G., Dvijotham, K., Vecerík, M., Hester, T., Păduraru, C., & Tassa, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.08757 · Cited by 250
Recent Advances in Parameteridentification Techniques for O.D.E.
Bock, H. G. (1983) · Birkhäuser Boston eBooks · doi:10.1007/978-1-4684-7324-7_7 · Cited by 245
Robust Constrained Learning-based NMPC enabling reliable mobile robot path tracking
Ostafew, C. J., Schoellig, A. P., & Barfoot, T. D. (2016) · The International Journal of Robotics Research · doi:10.1177/0278364916645661 · Cited by 232
Feasibility issues in linear model predictive control
Scokaert, P. O. M., & Rawlings, J. B. (1999) · AIChE Journal · doi:10.1002/aic.690450805 · Cited by 204
Differentiable MPC for End-to-end Planning and Control
Amos, B., Rodriguez, I. D. J., Sacks, J., Boots, B., & Kolter, J. Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.13400 · Cited by 189
Optimizing process economics online using model predictive control
Amrit, R., Rawlings, J. B., & Biegler, L. T. (2013) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2013.07.015 · Cited by 154
OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World
Pham, T.-H., Magistris, G. D., & Tachibana, R. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460547 · Cited by 140
Machine Learning Algorithms in Bipedal Robot Control
Wang, S., Chaovalitwongse, W. A., & Babuška, R. (2012) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2012.2186565 · Cited by 136
On Necessity and Robustness of Dissipativity in Economic Model Predictive Control
Müller, M. A., Angeli, D., & Allgöwer, F. (2014) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2014.2361193 · Cited by 126
Sensitivity Analysis and Real-Time Optimization of Parametric Nonlinear Programming Problems
Büskens, C., & Maurer, H. (2001) · Springer eBooks · doi:10.1007/978-3-662-04331-8_1 · Cited by 97
Grüne, L., & Guglielmi, R. (2018) · SIAM Journal on Control and Optimization · doi:10.1137/17m112350x · Cited by 82
Practical Reinforcement Learning of Stabilizing Economic MPC
Zanon, M., Gros, S., & Bemporad, A. (2019) · European Control Conference (ECC) · doi:10.23919/ecc.2019.8795816 · Cited by 54
Self-citationLearning-based Model Predictive Control for Safe Exploration and Reinforcement LearningPreprint
Koller, T., Berkenkamp, F., Turchetta, M., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.12189 · Cited by 48
A model for system uncertainty in reinforcement learning
Murray, R., & Palladino, M. (2018) · Systems & Control Letters · doi:10.1016/j.sysconle.2018.09.011 · Cited by 22
Sonntag, C., Stursberg, O., & Engell, S. (2006) · IFAC Proceedings Volumes · doi:10.3182/20060607-3-it-3902.00039 · Cited by 13
Sonntag, C., Stursberg, O., & Engell, S. (2006) · Elsevier eBooks · doi:10.1016/b978-008044613-4.50038-0 · Cited by 11
Cited by
Data-Driven Model Predictive Control With Stability and Robustness Guarantees
Berberich, J., Kohler, J., Muller, M. A., & Allgower, F. (2021) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2020.3000182 · Cited by 803
Wang, D., Gao, N., Liu, D., Li, J., & Lewis, F. L. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123843 · Cited by 364
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications
Gu, S., Yang, L., Du, Y., Chen, G., Walter, F., Wang, J., & Knoll, A. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3457538 · Cited by 315
Data-driven learning-based Model Predictive Control for energy-intensive systems
Chen, J., Xu, G., & Zhou, Z. (2023) · Advanced Engineering Informatics · doi:10.1016/j.aei.2023.102208 · Cited by 34
Metadata
- Authors
- Zanon, M., & Gros, S.
- Venue
- IEEE Transactions on Automatic Control
- Published
- 2021-08
- Volume
- 66
- Issue
- 8
- Pages
- 3638-3652
- ISSN
- 0018-9286 (ISSN-L)
- 1558-2523
- 2334-3303
- OpenAlex
- W2948652605
- Cited by
- 338
- References
- 40
- FWCI
- 17.88
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.