Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 5

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

Adversarial Training for High-Stakes Reliability

Ziegler, Daniel, Nix, Seraphina, Chan, Lawrence, Bauman, Tim, Schmidt-Nielsen, Peter, Lin, Tao, Scherlis, Adam, Nabeshima, Noa, Weinstein-Raun, Benjamin, Haas, Daniel De, Shlegeris, Buck, Thomas, Nate·2022·Advances in Neural Information Processing Systems 35
DOI 10.52202/068431-0674
Open this paper in CoChat — free account

Abstract

In the future, powerful AI systems may be deployed in high-stakes settings, where a single failure could be catastrophic. One technique for improving AI safety in high-stakes settings is adversarial training, which uses an adversary to generate examples to train on in order to achieve better worst-case performance. In this work, we used a safe language generation task (``avoid injuries'') as a testbed for achieving high reliability through adversarial training. We created a series of adversarial training techniques -- including a tool that assists human adversaries -- to find and eliminate failures in a classifier that filters text completions suggested by a generator. In our task, we determined that we can set very conservative classifier thresholds without significantly impacting the quality of the filtered outputs. We found that adversarial training increased robustness to the adversarial attacks that we trained on -- doubling the time for our contractors to find adversarial examples both with our tool (from 13 to 26 minutes) and without (from 20 to 44 minutes) -- without affecting in-distribution performance. We hope to see further work in the high-stakes reliability setting, including more powerful tools for enhancing human adversaries and better ways to measure high levels of reliability, until we can confidently rule out the possibility of catastrophic deployment-time failures of powerful models.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.52202/068431-0674

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 A representation of our adversarial training loop. Starting from an initial story dataset consisting of prompts and generator completions (Section 4.3), we trained a classifier to detect injurious completions. We then iteratively attacked our classifier using unaugmented humans (Section 4.4.1), automatically paraphrased previous adversarial examples (Section 4.4.2), and tool-assisted human rewrites (Section 4.4.3), while training on the resulting adversarial examples.
Figure 1 A representation of our adversarial training loop. Starting from an initial story dataset consisting of prompts and generator completions (Section 4.3), we trained a classifier to detect injurious completions. We then iteratively attacked our classifier using unaugmented humans (Section 4.4.1), automatically paraphrased previous adversarial examples (Section 4.4.2), and tool-assisted human rewrites (Section 4.4.3), while training on the resulting adversarial examples.CC-BY · From the PDF
Image 2
CC-BY · From the PDF
Figure 2 (Left:) The amount of time spent per successful tool-assisted adversarial example (higher is better) and (Right:) the egregiousness of said adversarial examples (lower is better), for each of the four classifiers. Error bars represent boot-strapped 95% confidence intervals. As discussed in Section 5.3, the difficulty of finding adversarial examples increased significantly with adversarial training. In addition, the adversarial examples against classifiers trained against a more diverse pool of adversarial attacks tended to be less egregious, suggesting increased robustness.
Figure 2 (Left:) The amount of time spent per successful tool-assisted adversarial example (higher is better) and (Right:) the egregiousness of said adversarial examples (lower is better), for each of the four classifiers. Error bars represent boot-strapped 95% confidence intervals. As discussed in Section 5.3, the difficulty of finding adversarial examples increased significantly with adversarial training. In addition, the adversarial examples against classifiers trained against a more diverse pool of adversarial attacks tended to be less egregious, suggesting increased robustness.CC-BY · From the PDF
FNR on snippets from tool-assisted attack against baseline
FNR on snippets from tool-assisted attack against baselineCC-BY · From the PDF
Image 5
CC-BY · From the PDF
Figure 3 False negative rate (lower is better) for each of our four classifiers on each tool-assisted dataset. Error bars represent the central 95% Bayesian credible interval with a β ( 1 , 1 ) prior. On examples targeting them, classifiers must have an FNR of 1.0 by definition (faded bars), since our tool only accepts examples that the classifier accepts. Otherwise, we see a steady trend of lower FNRs for later classifiers, suggesting increased adversarial robustness.
Figure 3 False negative rate (lower is better) for each of our four classifiers on each tool-assisted dataset. Error bars represent the central 95% Bayesian credible interval with a β ( 1 , 1 ) prior. On examples targeting them, classifiers must have an FNR of 1.0 by definition (faded bars), since our tool only accepts examples that the classifier accepts. Otherwise, we see a steady trend of lower FNRs for later classifiers, suggesting increased adversarial robustness.CC-BY · From the PDF
Figure 4 Our tool-assisted attack user interface discussed in Section 4.4.3. Given an original prompt completion pair, users could edit the prompt or completion manually, or use our saliency-backed token substitution and insertion tool to edit the prompt. You can access a demo at https://www. ttft.io/talk-to-filtered-transformer .
Figure 4 Our tool-assisted attack user interface discussed in Section 4.4.3. Given an original prompt completion pair, users could edit the prompt or completion manually, or use our saliency-backed token substitution and insertion tool to edit the prompt. You can access a demo at https://www. ttft.io/talk-to-filtered-transformer .CC-BY · From the PDF
Figure 5 Our labeling user interface.
Figure 5 Our labeling user interface.CC-BY · From the PDF
Figure 6 ROC curves for 20 thresholds in logarithmic space between 0 . 0001 and 0 . 1 . The points on the chart are the thresholds we chose for each classifier. We do not see a meaningful difference in the performance for our more adversarially-trained classifiers when evaluated on in-distribution data.
Figure 6 ROC curves for 20 thresholds in logarithmic space between 0 . 0001 and 0 . 1 . The points on the chart are the thresholds we chose for each classifier. We do not see a meaningful difference in the performance for our more adversarially-trained classifiers when evaluated on in-distribution data.CC-BY · From the PDF
Figure 7 Validation set false negative rate (lower is better) for each of our four classifiers. Unsurprisingly, each model has a very low false negative rate on the validation set for the datasets it was trained on.
Figure 7 Validation set false negative rate (lower is better) for each of our four classifiers. Unsurprisingly, each model has a very low false negative rate on the validation set for the datasets it was trained on.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 45

  1. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017. · arXiv
  2. Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-609, 2020.
  3. Amol Mandhane, Anton Zhernov, Maribeth Rauh, Chenjie Gu, Miaosen Wang, Flora Xue, Wendy Shang, Derek Pang, Rene Claus, Ching-Han Chiang, et al. Muzero with self- competition for rate control in vp9 video compression. arXiv preprint arXiv:2202.06626, 2022. · arXiv
  4. Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34, 2021.
  5. Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1373, 2016.
  6. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. · arXiv
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language mod- els are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
  8. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. · arXiv
  9. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. · arXiv
  10. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. · arXiv
  11. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. · arXiv
  12. Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Ox- ford, UK, 2014. ISBN 978-0-19-967811-2.
  13. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
  14. Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In Interna- tional conference on machine learning, pages 783-792. PMLR, 2019.
  15. Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
  16. Daniel Brown, Scott Niekum, and Marek Petrik. Bayesian robust optimization for imitation learning. Advances in Neural Information Processing Systems, 33:2479-2491, 2020.
  17. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. arXiv preprint arXiv:1412.6572, 2014. · arXiv
  18. Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and J Doug Tygar. Adversarial machine learning. In Proceedings of the 4th ACM workshop on Security and artificial intelligence, pages 43-58, 2011.
  19. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. · arXiv
  20. Jonathan Uesato, Brendan O'donoghue, Pushmeet Kohli, and Aaron Oord. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning, pages 5025-5034. PMLR, 2018.
  21. Aditi Raghunathan, Jacob Steinhardt, and Percy S Liang. Semidefinite relaxations for certify- ing robustness to adversarial examples. Advances in Neural Information Processing Systems, 31, 2018.
  22. Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pages 5286- 5295. PMLR, 2018.
  23. Mahmood Sharif, Lujo Bauer, and Michael K Reiter. On the suitability of lp-norms for creating and preventing adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1605-1613, 2018.
  24. Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Alek- sander Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
  25. T. B. Brown, N. Carlini, C. Zhang, C. Olsson, P. Christiano, and I. Goodfellow. Unrestricted adversarial examples. arXiv preprint arXiv:1809.08352, 2018. · arXiv
  26. Mingyang Yi, Lu Hou, Jiacheng Sun, Lifeng Shang, Xin Jiang, Qun Liu, and Zhiming Ma. Improved ood generalization via adversarial training and pretraing. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, vol- ume 139 of Proceedings of Machine Learning Research, pages 11987-11997. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/yi21a.html.
  27. Alexis Ross, Tongshuang Wu, Hao Peng, Matthew E Peters, and Matt Gardner. Tailor: Gener- ating and perturbing text with semantic controls. arXiv preprint arXiv:2107.07150, 2021. · arXiv
  28. Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. Improving question answering model robustness with synthetic adversarial data generation. arXiv preprint arXiv:2104.08678, 2021. · arXiv
  29. Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adver- sarial attacks against text transformers. arXiv preprint arXiv:2104.13733, 2021. · arXiv
  30. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022. · arXiv
  31. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. arXiv preprint arXiv:1908.06083, 2019. · arXiv
  32. Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. arXiv preprint arXiv:2111.02840, 2021. · arXiv
  33. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethink- ing benchmarking in nlp. arXiv preprint arXiv:2104.14337, 2021. · arXiv
  34. Eric Wallace, Adina Williams, Robin Jia, and Douwe Kiela. Analyzing dynamic adversarial training data in the limit. arXiv preprint arXiv:2110.08514, 2021. · arXiv
  35. Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering. Transactions of the Association for Computational Linguistics, 7:387-401, 2019.
  36. Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543, 2021. · arXiv
  37. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transform- ers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. · arXiv
  38. Chaitra Hegde and Shrikumar Patil. Unsupervised paraphrase generation using pre-trained language models. arXiv preprint arXiv:2006.05477, 2020. · arXiv
  39. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020. · arXiv
  40. Benedikt Boecking, Willie Neiswanger, Eric Xing, and Artur Dubrawski. Interactive weak supervision: Learning useful heuristics for data labeling. arXiv preprint arXiv:2012.06046, 2020. · arXiv
  41. Stephen Casper, Max Nadeau, Dylan Hadfield-Menell, and Gabriel Kreiman. Robust feature- level adversaries are interpretability tools, 2021. URL https://arxiv.org/abs/2110. 03605.
  42. Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. Distilling model fail- ures as directions in latent space. arXiv preprint arXiv:2206.14754, 2022. · arXiv
  43. Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends® in Robotics, 7(1-2):1-179, 2018.
  44. Stefan Webb, Tom Rainforth, Yee Whye Teh, and M Pawan Kumar. A statistical approach to assessing neural network robustness. arXiv preprint arXiv:1811.07209, 2018. · arXiv
  45. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. · arXiv

Cited by

See all 5

Metadata

Authors
Ziegler, D., Nix, S., Chan, L., Bauman, T., Schmidt-Nielsen, P., Lin, T., Scherlis, A., Nabeshima, N., Weinstein-Raun, B., Haas, D. D., Shlegeris, B., & Thomas, N.
Venue
Advances in Neural Information Processing Systems 35
Publisher
Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Published
2022
Pages
9274-9286
DOI
10.52202/068431-0674
Cited by
1
References
4

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.