Adversarial Policies: Attacking Deep Reinforcement Learning
Open accessAbstract
Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent's observations. This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent. Videos are available at https://adversarialpolicies.github.io/.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Seemingly benign agents might establish secret collusion channels through steganographic communication, engage in coordinated attacks that appear innocuous when viewed individually, or exploit information asymmetries to covertly manipulate shared environments or deceive other agents' decision processes.
“Seemingly benign agents might establish secret collusion channels through steganographic communication (Motwani et al., 2024b;Vaikuntanathan & Zamir, 2026), engage in coordinated attacks that appear innocuous when viewed individually (Davies et al., 2025), or exploit information asymmetries to covertly manipulate shared environments, such as markets or social media, or even directly deceive other”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Gleave et al., 2019
Split into 3 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Part of this paper’s reference list could not be split into single entries.
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018) · Science · doi:10.1126/science.aar6404 · Cited by 3,562 · cited 1× · see citing works
“Although it is known that self-play with deep RL may not converge, or converge only to a local rather than global Nash, self-play has been used with great success in a number of works focused on playing adversarial games directly against humans (Silver et al., 2018; OpenAI, 2018).”
BackgroundRobust adversarial reinforcement learning
Pinto, L., Davidson, J., Sukthankar, R., & Gupta, A. (2017) · arXiv · doi:10.48550/arxiv.1703.02702 · Cited by 507 · cited 1× · see citing works
“Prior work has also applied adversarial training to improve the robustness of deep RL policies, where the adversary exerts a force vector on the victim or varies dynamics parameters such as friction (Pinto et al., 2017; Mandlekar et al., 2017; Pattanaik et al., 2018).”
Method
Flagged first, then by citations
Research works printed but not linked: 11
- [2]2. David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech M. Czarnecki, Julien Pérolat, Max Jaderberg, and Thore Graepel. Open-ended learning in symmetric zero-sum games. arXiv:1901.08106v1 [cs.LG], 2019. · arXiv
- [3]3. Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. In Proceedings of the International Conference on Learning Representations (ICLR) , 2018a.
- [5]5. Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Advances in Neural Information Processing Systems (NeurIPS) , pages 2494-2504. 2018.
- [6]6. Vahid Behzadan and Arslan Munir. Adversarial reinforcement learning framework for benchmarking collision avoidance mechanisms in autonomous vehicles. IEEE Intelligent Transportation Systems Magazine , 2019.
- [7]7. Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret minimization. In Proceedings of the International Conference on Machine Learning (ICML) , 2019.
- [8]8. Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the Conference on Robot Learning (CoRL) , volume 78, pages 1-16, 2017.
- [9]9. John Doyle, James A. Primbs, Benjamin Shapiro, and Vesna Nevisti´ c. Nonlinear games: examples and counterexamples. In Proceedings of IEEE Conference on Decision and Control (CDC) , volume 4, pages 3915-3920, 1996.
- [10]10. Shromona Ghosh, Felix Berkenkamp, Gireeja Ranade, Shaz Qadeer, and Ashish Kapoor. Verifying controllers against adversarial examples with Bayesian optimization. In IEEE International Conference on Robotics and Automation (ICRA) , pages 7306-7313, 2018.
- [11]11. Justin Gilmer, Ryan P. Adams, Ian Goodfellow, David Andersen, and George E. Dahl. Motivating the rules of the game for adversarial example research. arXiv:1807.06732v2 [cs.LG], 2018a. · arXiv
- [12]12. Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow. Adversarial spheres. arXiv:1801.02774v3 [cs.CV], 2018b. · arXiv
- [13]13. I.J. Good. On the principle of total evidence. The British Journal for the Philosophy of Science , 17 (4):319-321, 1967.
Cited by
Sorted by citations
Single and multi-agent deep reinforcement learning for AI-enabled wireless networks: A tutorial
Feriani, A., & Hossain, E. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3063822 · Cited by 410 · see citing works
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
“Even “superhuman” policies can fail catastrophically against policies specifically designed to exploit them [Gleave et al., 2020b, Wu et al., 2021b, Wang et al., 2022].”
BackgroundOpen Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
de Witt, C. S., Krawiecka, K., Krawczuk, I., Hagag, B., Anderson, W. L., Belcak, P., Bucknall, B., Cai, X., Chopra, A., Cohen, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.02077 · Cited by 2 · see citing works
“…attacks that appear innocuous when viewed individually (Davies et al., 2025), or exploit information asymmetries to covertly manipulate shared environments, such as markets or social media, or even directly deceive other agents’ decision processes (Gleave et al., 2019; Franzmeyer et al., 2024).”
- Authors
- Gleave, A., Dennis, M., Wild, C., Kant, N., Levine, S., & Russell, S.
- Affiliations
- University of California, Berkeley (US)
- Nvidia (United Kingdom) (GB)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1905.10615
- OpenAlex
- W2945924974
- Cited by
- 200
- References
- 3
- Citations by year
- 2025: 7 · 2024: 22 · 2023: 35 · 2022: 38 · 2021: 56 · 2020: 39 · 2019: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Keywords
- deep reinforcement learning, adversarial perturbations, multi-agent games, zero-sum games, self-play, policy robustness, high-dimensional environments, neural network activation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
