Robust Reinforcement Learning: A Review of Foundations and Recent Advances
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Reinforcement learning (RL) has become a highly successful framework for learning in Markov decision processes (MDP). Due to the adoption of RL in realistic and complex environments, solution robustness becomes an increasingly important aspect of RL deployment. Nevertheless, current RL algorithms struggle with robustness to uncertainty, disturbances, or structural changes in the environment. We survey the literature on robust approaches to reinforcement learning and categorize these methods in four different ways: (i) Transition robust designs account for uncertainties in the system dynamics by manipulating the transition probabilities between states; (ii) Disturbance robust designs leverage external forces to model uncertainty in the system behavior; (iii) Action robust designs redirect transitions of the system by corrupting an agent’s output; (iv) Observation robust designs exploit or distort the perceived system state of the policy. Each of these robust designs alters a different aspect of the MDP. Additionally, we address the connection of robustness to the risk-based and entropy-regularized RL formulations. The resulting survey covers all fundamental concepts underlying the approaches to robust reinforcement learning and their recent advances.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149 · cited 2×
“To track the true state, the agent must maintain a belief that contains knowledge of past states to estimate which of the true states is represented by the observed state [97,98].”
BackgroundThe Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
Practical Black-Box Attacks against Machine Learning
Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2017) · Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security · doi:10.1145/3052973.3053009 · Cited by 3,495 · cited 1×
“Adversarial attacks are mainly known from input perturbations in deep learning [161,162].”
BackgroundMulti-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., & Mordatch, I. (2017) · arXiv · doi:10.48550/arxiv.1706.02275 · Cited by 3,377
Bellman, R. (1957) · Indiana University Mathematics Journal · doi:10.1512/iumj.1957.6.56038 · Cited by 2,408 · cited 11×
“The extension to robust MDPs is presented as the uncertainty robust Bellman equation (URBE) [140].”
BackgroundMethodInfluentialCounterfactual Multi-Agent Policy Gradients
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., & Whiteson, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11794 · Cited by 1,873
Robust optimization ? methodology and applications
Ben-Tal, A., & Nemirovski, A. (2002) · Mathematical Programming · doi:10.1007/s101070100286 · Cited by 1,409
Learning to Communicate with Deep Multi-Agent Reinforcement Learning
Foerster, J. N., Assael, Y. M., de Freitas, N., & Whiteson, S. (2016) · arXiv · doi:10.48550/arxiv.1605.06676 · Cited by 1,217
Multi-agent deep reinforcement learning: a survey
Gronauer, S., & Diepold, K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09996-w · Cited by 957 · cited 1×
“Hence, the stochastic game or Markov game is a natural extension of the reinforcement learning framework to multi-agent systems [95,113,114].”
BackgroundSimple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, R. J. (1992) · Machine Learning · doi:10.1023/a:1022672621406 · Cited by 817
Peters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738
Self-citationRobust adversarial reinforcement learning
Pinto, L., Davidson, J., Sukthankar, R., & Gupta, A. (2017) · arXiv · doi:10.48550/arxiv.1703.02702 · Cited by 507 · cited 9×
“[9] propose a two-player zero-sum Markov game between a protagonist π(·|s) and a destabilizing adversary π̄(·|s) defined by the tupleM := ( S ,A, Ā,P , r, γ ) .”
BackgroundMethodInfluentialRobust Markov Decision Processes
Wiesemann, W., Kuhn, D., & Rustem, B. (2013) · Mathematics of Operations Research · doi:10.1287/moor.1120.0566 · Cited by 312
A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning
Lanctot, M., Zambaldi, V., Gruslys, A., Lazaridou, A., Tuyls, K., Perolat, J., Silver, D., & Graepel, T. (2017) · arXiv · doi:10.48550/arxiv.1711.00832 · Cited by 295
Adversarially Robust Policy Learning: Active construction of physically-plausible perturbations
Mandlekar, A., Zhu, Y., Garg, A., Fei-Fei, L., & Savarese, S. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206245 · Cited by 116
Action robust reinforcement learning and applications in continuous control
Tessler, C., Efroni, Y., & Mannor, S. (2019) · arXiv · doi:10.48550/arxiv.1901.09184 · Cited by 108
Maximum Entropy RL (Provably) Solves Some Robust RL Problems
Eysenbach, B., & Levine, S. (2021) · arXiv · doi:10.48550/arxiv.2103.06257 · Cited by 69
Risk Averse Robust Adversarial Reinforcement Learning
Pan, X., Seita, D., Gao, Y., & Canny, J. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8794293 · Cited by 69
Cited by
Reinforcement learning for HVAC control in intelligent buildings: A technical and conceptual review
Al Sayed, K., Boodi, A., Sadeghian Broujeny, R., & Beddiar, K. (2024) · Journal of Building Engineering · doi:10.1016/j.jobe.2024.110085 · Cited by 140
Islam, S., Chiu, M.-H., Bonjour, T., de Oliveira, R., Bhargava, B., & Kejriwal, M. (2025) · IEEE Intelligent Systems · doi:10.1109/mis.2024.3469574 · Cited by 13
Directions of Technical Innovation for Regulatable AI Systems
Shen, X., Brown, H., Tao, J., Strobel, M., Tong, Y., Narayan, A., Soh, H., & Doshi-Velez, F. (2024) · Communications of the ACM · doi:10.1145/3653670 · Cited by 3
Metadata
- Authors
- Moos, J., Hansel, K., Abdulsamad, H., Stark, S., Clever, D., & Peters, J.
- Venue
- Machine Learning and Knowledge Extraction
- Publisher
- MDPI AG
- Published
- 2022-03-19
- Volume
- 4
- Issue
- 1
- Pages
- 276-315
- ISSN
- 2504-4990 (ISSN-L)
- OpenAlex
- W4221110788
- Cited by
- 127
- References
- 20
- FWCI
- 8.00
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.