Adversarially Robust Policy Learning: Active construction of physically-plausible perturbations
Abstract
Policy search methods in reinforcement learning have demonstrated success in scaling up to larger problems beyond toy examples. However, deploying these methods on real robots remains challenging due to the large sample complexity required during learning and their vulnerability to malicious intervention. We introduce Adversarially Robust Policy Learning (ARPL), an algorithm that leverages active computation of physically-plausible adversarial examples during training to enable robust policy learning in the source domain and robust performance under both random and adversarial input perturbations. We evaluate ARPL on four continuous control tasks and show superior resilience to changes in physical environment dynamics parameters and environment state as compared to state-of-the-art robust policy learning methods. Code, data, and additional experimental results are available at: stanfordvl.github.io/ARPL.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · see citing works
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · see citing works
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · see citing works
Adversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375 · see citing works
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1× · see citing works
“We used a curriculum learning approach [4] in order to train our agents on increasing φ Algorithm 1: Adversarially Robust Policy Learning”
MethodDeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · see citing works
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · see citing works
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578 · see citing works
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160 · see citing works
Universal Adversarial Perturbations
Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., & Frossard, P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.17 · Cited by 2,680 · see citing works
End-to-end training of deep visuomotor policies
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2016) · Journal of Machine Learning Research · Cited by 2,192 · see citing works
Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971 · see citing works
Evasion attacks against machine learning at test time
Biggio, B., Corona, I., Maiorca, D., Nelson, B. A., Srndic, N., Laskov, P., Giacinto, G., & Roli, F. (2013) · Cited by 1,658 · see citing works
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533 · see citing works
Self-citationTransfer Learning for Reinforcement Learning Domains: A Survey
Taylor, M. E., & Stone, P. (2009) · doi:10.5555/1577069.1755839 · Cited by 1,521 · see citing works
(1995) · Control Engineering Practice · doi:10.1016/0967-0661(95)90063-2 · Cited by 1,314 · see citing works
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1606.04671 · Cited by 1,307 · see citing works
(2015) · Journal of Machine Learning Research · Cited by 1,119 · see citing works
Evasion Attacks against Machine Learning at Test Time
Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G., & Roli, F. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40994-3_25 · Cited by 890 · see citing works
Can machine learning be secure?
Barreno, M., Nelson, B. A., Sears, R., Joseph, A. D., & Tygar, J. D. (2006) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/1128817.1128824 · Cited by 852 · see citing works
Robust Control of Markov Decision Processes with Uncertain Transition Matrices
Nilim, A., & Ghaoui, L. E. (2005) · Operations Research · doi:10.1287/opre.1050.0216 · Cited by 702 · see citing works
On the (Statistical) Detection of Adversarial ExamplesPreprint
Grosse, K., Manoharan, P., Papernot, N., Michael Backes, A., & McDaniel, P. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.06280 · Cited by 546 · see citing works
Sim-to-Real Robot Learning from Pixels with Progressive Nets
Rusu, A. A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1610.04286 · Cited by 385 · see citing works
Learning modular neural network policies for multi-task and multi-robot transfer
Devin, C., Gupta, A., Darrell, T., Abbeel, P., & Levine, S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989250 · Cited by 324 · see citing works
Practical Black-Box Attacks against Deep Learning Systems using Adversarial Examples.Preprint
Papernot, N., McDaniel, P. D., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2016) · arXiv (Cornell University) · Cited by 286 · see citing works
Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks
Behzadan, V., & Munir, A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-62416-7_19 · Cited by 247 · see citing works
Preparing for the unknown: Learning a universal policy with online system identification
Yu, W., Tan, J., Karen Liu, C., & Turk, G. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.048 · Cited by 237 · see citing works
Empagliflozin for Type 2 Diabetes Mellitus: An Overview of Phase 3 Clinical Trials
Levine, M. J. (2016) · Current Diabetes Reviews · doi:10.2174/1573399812666160613113556 · Cited by 235 · see citing works
Using inaccurate models in reinforcement learning
Abbeel, P., Quigley, M., & Ng, A. Y. (2006) · International Conference on Machine Learning · doi:10.1145/1143844.1143845 · Cited by 214 · see citing works
EPOpt: Learning Robust Neural Network Policies Using Model EnsemblesPreprint
Rajeswaran, A., Ghotra, S., Ravindran, B., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.01283 · Cited by 185 · see citing works
Thananjeyan, B., Garg, A., Krishnan, S., Chen, C., Miller, L., & Goldberg, K. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989275 · Cited by 144 · see citing works
Self-citationAdaPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems
Harrison, J., Garg, A., Ivanovic, B., Zhu, Y., Savarese, S., Fei-Fei, L., & Pavone, M. (2019) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-28619-4_34 · Cited by 21 · see citing works
Self-citationADAPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems
Harrison, J., Garg, A., Ivanovic, B., Zhu, Y., Savarese, S., Fei-Fei, L., & Pavone, M. (2017) · arXiv · doi:10.48550/arxiv.1707.04674 · Cited by 21
Self-citationLoss bounds for uncertain transition probabilities in Markov decision processes
Mastin, A., & Jaillet, P. (2012) · doi:10.1109/cdc.2012.6426504 · Cited by 14 · see citing works
He, C. (1997) · Numerische Mathematik · doi:10.1007/s002110050272 · Cited by 10 · see citing works
DeepMask: Masking DNN Models for robustness against adversarial samples.Preprint
Gao, J., Wang, B., & Qi, Y. (2017) · arXiv (Cornell University) · Cited by 8 · see citing works
Uncontrollability for Linear Autonomous Multi-input Dynamical Systems Depending on Parameters
Mailybaev, A. A. (2003) · SIAM Journal on Control and Optimization · doi:10.1137/s0363012902405339 · Cited by 7 · see citing works
The Distance to Uncontrollability via Linear Matrix Inequalities
Boyce, S. (2010) · VTechWorks (Virginia Tech) · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Salvato, E., Fenu, G., Medvet, E., & Pellegrino, F. A. (2021) · IEEE Access · doi:10.1109/access.2021.3126658 · Cited by 200 · see citing works
“…[43], [44], [45], [46], [47], [3], [8], [48] Motion control [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59] Locomotion [49], [50], [51], [60], [61], [62] When tackling a robot control problem with RL, according to the formulation and notation above provided, the designer…”
MethodInfluentialADAPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems
Harrison, J., Garg, A., Ivanovic, B., Zhu, Y., Savarese, S., Fei-Fei, L., & Pavone, M. (2017) · arXiv · doi:10.48550/arxiv.1707.04674 · Cited by 21
Self-citation
- Authors
- Mandlekar, A., Zhu, Y., Garg, A., Fei-Fei, L., & Savarese, S.
- Affiliations
- Stanford University (US)
- Venue
- 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Publisher
- IEEE
- Published
- 2017-09
- Pages
- 3932-3939
- ISSN
- 2153-0858 (ISSN-L)
- OpenAlex
- W2773691349
- Cited by
- 116
- References
- 40
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 12 · 2023: 15 · 2022: 12 · 2021: 28 · 2020: 21 · 2019: 16 · 2018: 5
- FWCI
- 5.24
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Keywords
- adversarial robustness, policy search, adversarial examples, continuous control, robot learning, robust policy learning, sample complexity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review