CoChat
Sign Up
PreprintCited by 10Adversarial Robustness in Machine LearningSecurity and Verification in ComputingSoftware Testing and Debugging Techniques

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Red Teaming with Mind Reading: White-Box Adversarial Policies Against RL Agents

Open access
Casper, Stephen, Killian, Taylor, Kreiman, Gabriel, Hadfield-Menell, Dylan·2022·arXiv
DOI 10.48550/arxiv.2209.02167arXiv 2209.02167
Open this paper in CoChat — free accountOpen PDF

Abstract

Adversarial examples can be useful for identifying vulnerabilities in AI systems before they are deployed. In reinforcement learning (RL), adversarial policies can be developed by training an adversarial agent to minimize a target agent's rewards. Prior work has studied black-box versions of these attacks where the adversary only observes the world state and treats the target agent as any other part of the environment. However, this does not take into account additional structure in the problem. In this work, we study white-box adversarial policies and show that having access to a target agent's internal state can be useful for identifying its vulnerabilities. We make two contributions. (1) We introduce white-box adversarial policies where an attacker observes both a target's internal state and the world state at each timestep. We formulate ways of using these policies to attack agents in 2-player games and text-generating language models. (2) We demonstrate that these policies can achieve higher initial and asymptotic performance against a target agent than black-box controls. Code is available at https://github.com/thestephencasper/lm_white_box_attacks

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2209.02167

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Black-box access to a model is sufficient for many adversarial policy attack algorithms, though white-box access enables even stronger exploits.

    “Black-box access to a model (e.g., via API access) is sufficient for many adversarial policy attack algorithms, though white-box access (enabled for example by open-sourced or leaked model weights) enables even stronger exploits (Kos and Song, 2017;Casper et al., 2022).”

    From Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023) · cites this paper as Casper et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Robust adversarial reinforcement learning

    Pinto, L., Davidson, J., Sukthankar, R., & Gupta, A. (2017) · arXiv · doi:10.48550/arxiv.1703.02702 · Cited by 507 · cited 3×

    “However, we successfully replicate the results from [45] and [58] that agents trained with RARL are more robust to distribution shift, and we find that the same is true of agents trained with a white-box version of RARL.”
    BackgroundMethod

Flagged first, then by citations

Research works printed but not linked: 53

  1. Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. arXiv preprint arXiv:1710.03748, 2017. · arXiv
  2. Vahid Behzadan and William Hsu. Adversarial exploitation of policy imitation. arXiv preprint arXiv:1906.01121, 2019. · arXiv
  3. Vahid Behzadan and William Hsu. Rl-based method for benchmarking the adversarial re- silience and robustness of deep reinforcement learning policies. In International Conference on Computer Safety, Reliability, and Security, pages 314-325. Springer, 2019.
  4. Siddhant Bhambri, Sumanyu Muku, Avinash Tulasi, and Arun Balaji Buduru. A survey of black-box adversarial attacks on computer vision models. arXiv preprint arXiv:1912.01667, 2019. · arXiv
  5. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016. · arXiv
  6. Stephen Casper. Achilles heels for agi/asi via decision theoretic adversaries. arXiv preprint arXiv:2010.05418, 2020. · arXiv
  7. Andrew Critch. A parametric, resource-bounded generalization of löb's theorem, and a robust cooperation criterion for open-source game theory. The Journal of Symbolic Logic, 84(4): 1368-1381, 2019.
  8. Andrew Critch, Michael Dennis, and Stuart Russell. Cooperative and uncooperative institution designs: Surprises and problems in open-source game theory. arXiv preprint arXiv:2208.07006, 2022. · arXiv
  9. Pavel Czempin and Adam Gleave. Reducing exploitability with population based training. arXiv preprint arXiv:2208.05083, 2022. · arXiv
  10. Abram Demski and Scott Garrabrant. Embedded agency. arXiv preprint arXiv:1902.09469, 2019. · arXiv
  11. Yaser Faghan, Nancirose Piazza, Vahid Behzadan, and Ali Fathi. Adversarial attacks on deep algorithmic trading policies. arXiv preprint arXiv:2010.11388, 2020. · arXiv
  12. Gregory Falco, Ben Shneiderman, Julia Badger, Ryan Carrier, Anton Dahbura, David Danks, Martin Eling, Alwyn Goodloe, Jerry Gupta, Christopher Hart, et al. Governing ai safety through independent audits. Nature Machine Intelligence, 3(7):566-571, 2021.
  13. Ted Fujimoto, Timothy Doster, Adam Attarian, Jill Brandenberger, and Nathan Hodas. Reward- free attacks in multi-agent reinforcement learning. arXiv preprint arXiv:2112.00940, 2021. · arXiv
  14. Adam Gleave, Michael Dennis, Neel Kant, Cody Wild, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv:1905.10615, 2019. · arXiv
  15. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.6572, 2014. · arXiv
  16. Jun Guo, Yonghong Chen, Yihang Hao, Zixin Yin, Yin Yu, and Simin Li. Towards comprehen- sive testing on the robustness of cooperative multi-agent reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 115-122, 2022.
  17. Wenbo Guo, Xian Wu, Sui Huang, and Xinyu Xing. Adversarial policy learning in two-player competitive games. In International Conference on Machine Learning, pages 3910-3919. PMLR, 2021.
  18. Joseph Y Halpern and Rafael Pass. Game theory with translucent players. International Journal of Game Theory, 47(3):949-976, 2018.
  19. He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep reinforcement learning. In International conference on machine learning, pages 1804-1813. PMLR, 2016.
  20. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
  21. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020. · arXiv
  22. Bairu Hou, Jinghan Jia, Yihua Zhang, Guanhua Zhang, Yang Zhang, Sijia Liu, and Shiyu Chang. Textgrad: Advancing robustness evaluation in nlp by gradient-driven optimization. arXiv preprint arXiv:2212.09254, 2022. · arXiv
  23. Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv preprint arXiv:1905.02175, 2019. · arXiv
  24. Ezgi Korkmaz. Adversarially trained neural policies in the fourier domain. In ICML 2021 Workshop on Adversarial Machine Learning, 2021.
  25. Ezgi Korkmaz. Investigating vulnerabilities of deep neural policies. In Uncertainty in Artificial Intelligence, pages 1661-1670. PMLR, 2021.
  26. Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXiv preprint arXiv:1705.06452, 2017. · arXiv
  27. Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michał Zaj ąc, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4501-4510, 2020.
  28. Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994, 2020. · arXiv
  29. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. · arXiv
  30. Alan J Lockett, Charles L Chen, and Risto Miikkulainen. Evolving explicit opponent models in game playing. In Proceedings of the 9th annual conference on Genetic and evolutionary computation, pages 2106-2113, 2007.
  31. Björn Lütjens, Michael Everett, and Jonathan P How. Certified adversarial robustness for deep reinforcement learning. In Conference on Robot Learning, pages 1328-1337. PMLR, 2020.
  32. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. · arXiv
  33. Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, pages 663-670, 2000.
  34. Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui-Wei Weng. Robust deep reinforcement learning through adversarial loss. Advances in Neural Information Processing Systems, 34, 2021.
  35. Xinlei Pan, Daniel Seita, Yang Gao, and John Canny. Risk averse robust adversarial reinforce- ment learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 8522-8528. IEEE, 2019.
  36. Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learn- ing: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277, 2016. · arXiv
  37. Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632, 2017. · arXiv
  38. Alberto Pozanco, Susana Fernández, Daniel Borrajo, et al. Anticipatory counterplanning. arXiv preprint arXiv:2203.16171, 2022. · arXiv
  39. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  40. Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http://jmlr.org/papers/v22/ 20-1364.html.
  41. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http://arxiv.org/ abs/1908.10084. · arXiv
  42. Alessio Russo and Alexandre Proutiere. Optimal attacks on reinforcement learning policies. arXiv preprint arXiv:1907.13548, 2019. · arXiv
  43. Lucas Schott, Manon Césaire, Hatem Hajri, and Sylvain Lamprier. Improving robustness of deep reinforcement learning agents: Environment attacks based on critic networks. arXiv preprint arXiv:2104.03154, 2021. · arXiv
  44. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. · arXiv
  45. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. · arXiv
  46. Kai Liang Tan, Yasaman Esfandiari, Xian Yeow Lee, Soumik Sarkar, et al. Robustifying reinforcement learning agents via action space adversarial training. In 2020 American control conference (ACC), pages 3959-3964. IEEE, 2020.
  47. Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pages 6215-6224. PMLR, 2019.
  48. Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint arXiv:1704.03453, 2017. · arXiv
  49. Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint arXiv:2008.01825, 2020. · arXiv
  50. Xian Wu, Wenbo Guo, Hua Wei, and Xinyu Xing. Adversarial policy training against deep reinforcement learning. In 30th USENIX Security Symposium (USENIX Security 21), pages 1883-1900, 2021.
  51. Kaiqing Zhang, Bin Hu, and Tamer Basar. On the stability and convergence of robust adver- sarial reinforcement learning: A case study on linear quadratic systems. Advances in Neural Information Processing Systems, 33:22056-22068, 2020.
  52. Bin Zhu, Zhaoquan Gu, Le Wang, Jinyin Chen, and Qi Xuan. Improving robustness of language models from a geometry-aware perspective. arXiv preprint arXiv:2204.13309, 2022. · arXiv
  53. Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. arXiv preprint arXiv:1909.11764, 2019. · arXiv

Cited by

Sorted by citations

  • Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

    Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101

    “Black-box access to a model (e.g., via API access) is sufficient for many adversarial policy attack algorithms, though white-box access (enabled for example by open-sourced or leaked model weights) enables even stronger exploits [Kos and Song, 2017, Casper et al., 2022].”
    BackgroundSelf-citation

Metadata

Authors
Casper, S., Killian, T., Kreiman, G., & Hadfield-Menell, D.
Affiliations
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
  • Boston Children's Hospital (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2209.02167
arXiv
2209.02167
OpenAlex
W4294962564
Cited by
10
References
1
Citations by year
2025: 3 · 2024: 5 · 2023: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
adversarial examples, theory of mind, black-box attacks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review