CoChat
Sign Up
PreprintCited by 13

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Baseline Defenses for Adversarial Attacks Against Aligned Language Models

Jain, Neel, Schwarzschild, Avi, Wen, Yuxin, Somepalli, Gowthami, Kirchenbauer, John, Chiang, Ping-yeh, Goldblum, Micah, Saha, Aniruddha, Geiping, Jonas, Goldstein, Tom·2023·arXiv
DOI 10.48550/arxiv.2309.00614arXiv 2309.00614
Open this paper in CoChat — free account

Abstract

As Large Language Models quickly become ubiquitous, it becomes critical to understand their security vulnerabilities. Recent work shows that text optimizers can produce jailbreaking prompts that bypass moderation and alignment. Drawing from the rich body of work on adversarial machine learning, we approach these attacks with three questions: What threat models are practically useful in this domain? How do baseline defense techniques perform in this new domain? How does LLM security differ from computer vision? We evaluate several baseline defense strategies against leading adversarial attacks on LLMs, discussing the various settings in which each is feasible and effective. Particularly, we look at three types of defenses: detection (perplexity based), input preprocessing (paraphrase and retokenization), and adversarial training. We discuss white-box and gray-box settings and discuss the robustness-performance trade-off for each of the defenses considered. We find that the weakness of existing discrete optimizers for text, combined with the relatively high costs of optimization, makes standard adaptive attacks more challenging for LLMs. Future research will be needed to uncover whether more powerful optimizers can be developed, or whether the strength of filtering and preprocessing defenses is greater in the LLMs domain than it has been in computer vision.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2309.00614

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 66

  1. Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. In Proceedings of the 35th In- ternational Conference on Machine Learning, pp. 274-283. PMLR, July 2018. URL https: //proceedings.mlr.press/v80/athalye18a.html.
  2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. · arXiv
  3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b. · arXiv
  4. Arjun Nitin Bhagoji, Daniel Cullina, Chawin Sitawarin, and Prateek Mittal. Enhancing robustness of machine learning systems via data transformations. In 2018 52nd Annual Conference on Infor- mation Sciences and Systems (CISS), pp. 1-5. IEEE, 2018.
  5. Nicholas Carlini and David Wagner. Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AISec '17, pp. 3-14, New York, NY, USA, November 2017. Association for Computing Machinery. ISBN 978-1-4503-5202-4. doi: 10.1145/3128572.3140444. URL https://dl. acm.org/doi/10.1145/3128572.3140444. · DOI · Paper page
  6. Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On Evaluating Adversarial Robustness. arxiv:1902.06705[cs, stat], February 2019. doi: 10.48550/arXiv.1902.06705. URL http://arxiv.org/abs/1902.06705. · DOI
  7. Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? arxiv:2306.15447[cs], June 2023. doi: 10. 48550/arXiv.2306.15447. URL http://arxiv.org/abs/2306.15447. · DOI
  8. Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. Explore, Establish, Exploit: Red Teaming Language Models from Scratch. arxiv:2306.09442[cs], June 2023. doi: 10.48550/arXiv.2306.09442. URL http://arxiv.org/abs/2306.09442. · DOI
  9. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023. · arXiv
  10. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023. · arXiv
  11. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. HotFlip: White-box adversarial ex- amples for text classification. In Proceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 2: Short Papers), pp. 31-36, Melbourne, Aus- tralia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-2006. URL https://aclanthology.org/P18-2006. · DOI
  12. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022. · arXiv
  13. Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pp. 50-56. IEEE, 2018.
  14. Yue Gao, I. Shumailov, Kassem Fawaz, and Nicolas Papernot. On the Limita- tions of Stochastic Pre-processing Defenses. In Advances in Neural Informa- tion Processing Systems, volume 35, pp. 24280-24294, December 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/ 997089469acbeb410405e43f0011be1f-Abstract-Conference.html.
  15. Amin Ghiasi, Ali Shafahi, and Tom Goldstein. Breaking certified defenses: Semantic adversarial examples with spoofed robustness certificates. In International Conference on Learning Repre- sentations, 2019.
  16. Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Mari- beth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022. · arXiv
  17. Micah Goldblum, Avi Schwarzschild, Ankit Patel, and Tom Goldstein. Adversarial attacks on ma- chine learning systems for high-frequency trading. In Proceedings of the Second ACM Interna- tional Conference on AI in Finance, pp. 1-9, 2021.
  18. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. · arXiv
  19. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arxiv:2302.12173[cs], May 2023. doi: 10.48550/arXiv.2302.12173. URL http://arxiv.org/abs/2302.12173. · DOI · Paper page
  20. Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel. On the (Statistical) Detection of Adversarial Examples. arxiv:1702.06280[cs, stat], October 2017. doi: 10.48550/arXiv.1702.06280. URL http://arxiv.org/abs/1702.06280. · DOI
  21. Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint arXiv:1412.5068, 2014. · arXiv
  22. Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based Adversarial Attacks against Text Transformers. arxiv:2104.13733[cs], April 2021. doi: 10.48550/arXiv.2104. 13733. URL http://arxiv.org/abs/2104.13733. · DOI
  23. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved Problems in ML Safety. arxiv:2109.13916[cs], June 2022. doi: 10.48550/arXiv.2109.13916. URL http: //arxiv.org/abs/2109.13916. · DOI · Paper page
  24. Hossein Hosseini and Radha Poovendran. Semantic adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1614-1619, 2018.
  25. Neel Jain, Khalid Saifullah, Yuxin Wen, John Kirchenbauer, Manli Shu, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Bring your own data! self-supervised evaluation for large language models. arXiv preprint arXiv:2306.13651, 2023. · arXiv
  26. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. arxiv:2302.05733[cs], February 2023. doi: 10.48550/arXiv.2302.05733. URL http:// arxiv.org/abs/2302.05733. · DOI · Paper page
  27. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of water- marks for large language models. arXiv preprint arXiv:2306.04634, 2023. · arXiv
  28. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human prefer- ences. In International Conference on Machine Learning, pp. 17506-17533. PMLR, 2023.
  29. Taku Kudo. Subword regularization: Improving neural network translation models with multi- ple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 66-75, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1007. URL https: //aclanthology.org/P18-1007. · DOI · Paper page
  30. Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. Multi- step Jailbreaking Privacy Attacks on ChatGPT. arxiv:2304.05197[cs], May 2023. doi: 10.48550/ arXiv.2304.05197. URL http://arxiv.org/abs/2304.05197. · DOI
  31. Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. Textbugger: Generating adversarial text against real-world applications. arXiv preprint arXiv:1812.05271, 2018. · arXiv
  32. Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. BERT-ATTACK: Ad- versarial attack against BERT using BERT. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 6193-6202, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.500. URL https://aclanthology.org/2020.emnlp-main.500. · DOI
  33. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. · arXiv
  34. Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai- Wei Chang, Aram Galstyan, and Rahul Gupta. FLIRT: Feedback Loop In-context Red Teaming. arxiv:2308.04265[cs], August 2023. doi: 10.48550/arXiv.2308.04265. URL http://arxiv. org/abs/2308.04265. · DOI
  35. Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp. 135-147, 2017a.
  36. Dongyu Meng and Hao Chen. MagNet: A Two-Pronged Defense against Adversarial Exam- ples. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communica- tions Security, CCS '17, pp. 135-147, New York, NY, USA, October 2017b. Association for Computing Machinery. ISBN 978-1-4503-4946-8. doi: 10.1145/3133956.3134057. URL https://dl.acm.org/doi/10.1145/3133956.3134057. · DOI
  37. Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprint arXiv:1702.04267, 2017. · arXiv
  38. Han Cheol Moon, Shafiq Joty, Ruochen Zhao, Megh Thakkar, and Xu Chi. Randomized smoothing with masked inference for adversarially robust text classifications. arXiv preprint arXiv:2305.06522, 2023. · arXiv
  39. John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP. arXiv:2005.05909 [cs], October 2020. URL http://arxiv.org/abs/2005.05909. arXiv: 2005.05909. · arXiv
  40. Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Animashree Anand- kumar. Diffusion Models for Adversarial Purification. In Proceedings of the 39th Interna- tional Conference on Machine Learning, pp. 16805-16827. PMLR, June 2022. URL https: //proceedings.mlr.press/v162/nie22a.html.
  41. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
  42. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023. URL https://arxiv.org/abs/2306.01116. · arXiv
  43. Fábio Perez and Ian Ribeiro. Ignore Previous Prompt: Attack Techniques For Language Mod- els. arxiv:2211.09527[cs], November 2022. doi: 10.48550/arXiv.2211.09527. URL http: //arxiv.org/abs/2211.09527. · DOI · Paper page
  44. Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. Bpe-dropout: Simple and effective subword regularization. arXiv preprint arXiv:1910.13267, 2019. · arXiv
  45. Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. Visual Adver- sarial Examples Jailbreak Aligned Large Language Models. In The Second Workshop on New Frontiers in Adversarial Machine Learning, August 2023. URL https://openreview. net/forum?id=cZ4j7L6oui.
  46. Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. Data Augmentation Can Improve Robustness. In Advances in Neural Information Processing Systems, volume 34, pp. 29935-29948.
  47. Curran Asso- ciates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ fb4c48608ce8825b558ccf07169a3421-Abstract.html.
  48. Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint arXiv:1805.06605, 2018. · arXiv
  49. Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Advances in Neural Information Processing Systems, volume 32.
  50. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/ hash/7503cfacd12053d309b6bed5c89de212-Abstract.html.
  51. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825, 2023. · arXiv
  52. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980, 2020. · arXiv
  53. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. · arXiv
  54. Florian Tramer. Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Them. In Pro- ceedings of the 39th International Conference on Machine Learning, pp. 21692-21702. PMLR, June 2022. URL https://proceedings.mlr.press/v162/tramer22a.html.
  55. Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On Adaptive Attacks to Adversarial Example Defenses. In Advances in Neural In- formation Processing Systems, volume 33, pp. 1633-1645.
  56. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 11f38f8ecd71867b42433548d1078e38-Abstract.html.
  57. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adver- sarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pp. 2153-2162, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1221. URL https://aclanthology.org/D19-1221. · DOI · Paper page
  58. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. · arXiv
  59. Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Dis- covery. arXiv preprint arXiv:2302.03668, February 2023. URL https://arxiv.org/abs/ 2302.03668v1. · arXiv
  60. Zuxuan Wu, Ser-Nam Lim, Larry S Davis, and Tom Goldstein. Making an invisibility cloak: Real world adversarial attacks on object detectors. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 1-17. Springer, 2020.
  61. Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems, 30(9):2805-2824, 2019.
  62. Zhen Zhang, Guanhua Zhang, Bairu Hou, Wenqi Fan, Qing Li, Sijia Liu, Yang Zhang, and Shiyu Chang. Certified robustness for large language models with self-denoising. arXiv preprint arXiv:2307.07171, 2023. · arXiv
  63. Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. arXiv preprint arXiv:1909.11764, 2019. · arXiv
  64. Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, and Xing Xie. PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. arxiv:2306.04528[cs], June 2023. doi: 10.48550/arXiv.2306.04528. URL http://arxiv.org/abs/2306.04528. · DOI · Paper page
  65. Roland S. Zimmermann, Wieland Brendel, Florian Tramer, and Nicholas Carlini. In- creasing Confidence in Adversarial Robustness Evaluations. In Advances in Neural Information Processing Systems, volume 35, pp. 13174-13189, December 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/ 5545d9bcefb7d03d5ad39a905d14fbe3-Abstract-Conference.html.
  66. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models. arxiv:2307.15043[cs], July 2023. doi: 10.48550/arXiv. 2307.15043. URL http://arxiv.org/abs/2307.15043. · DOI · Paper page

Cited by

Sorted by citations

See all 13

Metadata

Authors
Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., Chiang, P.-Y., Goldblum, M., Saha, A., Geiping, J., & Goldstein, T.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2309.00614
arXiv
2309.00614
References
1
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review