CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record; 4 of the 4 references it cites were found in a registry or an index.

Gradient Routing: Masking Gradients to Localize Computation in Neural Networks

Cloud, Alex, Goldman-Wetzler, Jacob, Wybitul, Evžen, Miller, Joseph, Turner, Alexander Matt·2024·arXiv
DOI 10.48550/arxiv.2410.04332arXiv 2410.04332
Open this paper in CoChat — free account

Abstract

Neural networks are trained primarily based on their inputs and outputs, without regard for their internal mechanisms. These neglected mechanisms determine properties that are critical for safety, like (i) transparency; (ii) the absence of sensitive information or harmful capabilities; and (iii) reliable generalization of goals beyond the training distribution. To address this shortcoming, we introduce gradient routing, a training method that isolates capabilities to specific subregions of a neural network. Gradient routing applies data-dependent, weighted masks to gradients during backpropagation. These masks are supplied by the user in order to configure which parameters are updated by which data points. We show that gradient routing can be used to (1) learn representations which are partitioned in an interpretable way; (2) enable robust unlearning via ablation of a pre-specified network subregion; and (3) achieve scalable oversight of a reinforcement learner by localizing modules responsible for different behaviors. Throughout, we find that gradient routing localizes capabilities even when applied to a limited, ad-hoc subset of the data. We conclude that the approach holds promise for challenging, real-world applications where quality data are scarce.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.04332

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Gradient routing applies weighted masks to selectively block or re-weight gradients during backpropagation. By supplying different masks for different data, the user can induce specialization in network subregions. The figure shows three masks, which would correspond to three data points.
Figure 1 Gradient routing applies weighted masks to selectively block or re-weight gradients during backpropagation. By supplying different masks for different data, the user can induce specialization in network subregions. The figure shows three masks, which would correspond to three data points.CC-BY · From the PDF
(b) Average (across 20 runs) validation set reconstruction losses, measured as the pixel-wise mean absolute error (MAE) for the Decoder and the certificates, demonstrating successful localization of information about digits. Run-to-run variation is negligible.
(b) Average (across 20 runs) validation set reconstruction losses, measured as the pixel-wise mean absolute error (MAE) for the Decoder and the certificates, demonstrating successful localization of information about digits. Run-to-run variation is negligible.CC-BY · From the PDF
Figure 3 Backpropagation in the Route step of Expand-Route-Ablate, showing the flow of gradients through a Transformer for tokens in the forget set. This assumes a learning rate of zero for the original dimensions in target layers. Gradients for retain tokens are unmodified. Additional dimensions, shown with dashed outlines, were added to target layers in the MLP and attention blocks, and will be removed after training in the Ablate step. All modules participate in the forward pass.
Figure 3 Backpropagation in the Route step of Expand-Route-Ablate, showing the flow of gradients through a Transformer for tokens in the forget set. This assumes a learning rate of zero for the original dimensions in target layers. Gradients for retain tokens are unmodified. Additional dimensions, shown with dashed outlines, were added to target layers in the MLP and attention blocks, and will be removed after training in the Ablate step. All modules participate in the forward pass.CC-BY · From the PDF
Figure 4 Effect of unlearning methods on forget and retain validation loss depending on the proportion of forget samples labeled. Highlighted regions denote 95% C.I. for the mean across at least N = 5 training runs. Left : how much each method increases forget loss after it is applied. For ERA and DEMix + ablate, this is pre- vs. post-ablation. Center : how much forget loss increases after a method is applied and the model is fine-tuned on 64 forget stories. (The minimum validation forget loss over fine-tuning is reported.) Right : the retain set performance after applying each method. Note: we include an additional data point for RMU at 0.95 of forget stories labeled. We also include a point for ERA+RMU (denoted with a '+') at full labeling.
Figure 4 Effect of unlearning methods on forget and retain validation loss depending on the proportion of forget samples labeled. Highlighted regions denote 95% C.I. for the mean across at least N = 5 training runs. Left : how much each method increases forget loss after it is applied. For ERA and DEMix + ablate, this is pre- vs. post-ablation. Center : how much forget loss increases after a method is applied and the model is fine-tuned on 64 forget stories. (The minimum validation forget loss over fine-tuning is reported.) Right : the retain set performance after applying each method. Note: we include an additional data point for RMU at 0.95 of forget stories labeled. We also include a point for ERA+RMU (denoted with a '+') at full labeling.CC-BY · From the PDF
Image 5
CC-BY · From the PDF
Image 6
CC-BY · From the PDF
Figure 6 The top half certificate reconstructions corresponding to fig. 2a, showing that the top half of the encoding contains information necessary to accurately reconstruct digits 0-4 while containing practically no information relevant to reconstructing digits 5-9.
Figure 6 The top half certificate reconstructions corresponding to fig. 2a, showing that the top half of the encoding contains information necessary to accurately reconstruct digits 0-4 while containing practically no information relevant to reconstructing digits 5-9.CC-BY · From the PDF
(b) Validation set reconstruction losses, measured as the pixel-wise mean absolute error (MAE) for the Decoder and the certificates.
(b) Validation set reconstruction losses, measured as the pixel-wise mean absolute error (MAE) for the Decoder and the certificates.CC-BY · From the PDF
ResNet classifier performance with and without gradient routing
ResNet classifier performance with and without gradient routingCC-BY · From the PDF
Figure 9 Retrained validation forget loss (i) before unlearning, and (ii) after applying unlearning, retraining on 64 forget stories, and taking the lowest validation forget set loss. The differences in these curves are displayed in the center panel of fig. 4.
Figure 9 Retrained validation forget loss (i) before unlearning, and (ii) after applying unlearning, retraining on 64 forget stories, and taking the lowest validation forget set loss. The differences in these curves are displayed in the center panel of fig. 4.CC-BY · From the PDF
Figure 10 The average (across runs) validation forget set loss for the ERA model and pure model over 40 steps of fine-tuning on batches of varying numbers of forget data points: 4, 16, and 64.
Figure 10 The average (across runs) validation forget set loss for the ERA model and pure model over 40 steps of fine-tuning on batches of varying numbers of forget data points: 4, 16, and 64.CC-BY · From the PDF
Figure 11 Average forget and retain set validation loss after training, after ablation, and after finetuning for ERA vs. a control. The control is the same as ERA except gradient routing is not applied. Note: the x -axis is not to scale; pre-ablation training is on 400,000 stories, ablation is immediate, and fine-tuning is on 64 stories.
Figure 11 Average forget and retain set validation loss after training, after ablation, and after finetuning for ERA vs. a control. The control is the same as ERA except gradient routing is not applied. Note: the x -axis is not to scale; pre-ablation training is on 400,000 stories, ablation is immediate, and fine-tuning is on 64 stories.CC-BY · From the PDF
Figure 12 Learning curves showing the ground-truth performance of each algorithm over the course of training at 10% oversight. Dark highlights correspond to approximate 95% confidence interval for the mean. Light highlights correspond to 5th and 95th quantiles.
Figure 12 Learning curves showing the ground-truth performance of each algorithm over the course of training at 10% oversight. Dark highlights correspond to approximate 95% confidence interval for the mean. Light highlights correspond to 5th and 95th quantiles.CC-BY · From the PDF
Figure 13 Ground truth returns comparing to two baselines, one without gradient routing, and the other with the gate module set to output a constant 0.5 (mixing the two experts equally). Dark highlights correspond to approximate 95% confidence interval for the mean (across multiple runs). Light highlights correspond to 5th and 95th quantiles.
Figure 13 Ground truth returns comparing to two baselines, one without gradient routing, and the other with the gate module set to output a constant 0.5 (mixing the two experts equally). Dark highlights correspond to approximate 95% confidence interval for the mean (across multiple runs). Light highlights correspond to 5th and 95th quantiles.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 4 references

Research works printed but not linked: 124

  1. Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language mod- els. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: //openreview.net/forum?id=XfHWcNTSHp. Survey Certification.
  2. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. · arXiv
  3. Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric J Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Chenyu Zhang, Ruiqi Zhong, Sean O hEigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Ed- wards, Aleksandar Petrov, Christian Schroeder de Witt, Sumeet Ramesh Motwani, Yoshua Ben- gio, Danqi Chen, Philip Torr, Samuel Albanie, Tegan Maharaj, Jakob Nicolaus Foerster, Florian Tramèr, He He, Atoosa Kasirzadeh, Yejin Choi, and David Krueger. Foundational challenges in assuring alignment and safety of large language models. Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id= oVTkOs8Pka. Survey Certification, Expert Certification.
  4. Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. ArXiv, abs/2307.15936, 2023. URL https://api.semanticscholar.org/ CorpusID:260334352. · arXiv
  5. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, John Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, E Perez, Jamie Kerr, Jared Mueller, Jeff Ladish, J Landau, Kamal Ndousse, Kamil Lukosit, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noem'i Mercado, Nova Dassarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Sam Bowman, Zac Hatfield-Dodds, Benjamin Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom B. Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback. ArXiv, abs/2212.08073, 2022. URL https://api.semanticscholar.org/ CorpusID:254823489. · arXiv
  6. Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24654, 2022.
  7. Ankur Bapna and Orhan Firat. Simple, scalable adaptation for neural machine translation. In Ken- taro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pp. 1538-1548, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1165. URL https://aclanthology.org/D19-1165. · DOI
  8. Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023. URL https://openreview.net/ forum?id=awIpKpwTwF.
  9. Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine- tuning for transformer-based masked language-models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1-9, Dublin, Ireland, May 2022. As- sociation for Computational Linguistics. doi: 10.18653/v1/2022.acl-short.1. URL https: //aclanthology.org/2022.acl-short. · DOI · Paper page
  10. Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
  11. John Beverley, David Limbaugh, Eric Merrell, Peter M. Koch, and Barry Smith. Capabilities: An ontology. In Proceedings of the Joint Ontology Workshops (JOWO) -Episode X: The Tukker Zomer of Ontology, and satellite events co-located with the 14th International Conference on Formal Ontology in Information Systems (FOIS 2024), Enschede, The Netherlands, July 15-19 2024. JOWO. URL https://arxiv.org/pdf/2405.00183. https://arxiv.org/ pdf/2405.00183. · arXiv
  12. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Neural Information Processing Systems, 2016. URL https://api.semanticscholar. org/CorpusID:1704893.
  13. Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159, 2021. doi: 10.1109/SP40001.2021.00019. · DOI · Paper page
  14. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbren- ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learn- ing, volume 235 of Proceedings of Machine Learning Research, pp. 4971-5012. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr.press/v235/burns24b.html.
  15. Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, pp. 463-480, 2015. doi: 10.1109/SP.2015. · DOI
  16. Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Neural Information Processing Systems, 2016. URL https://api.semanticscholar. org/CorpusID:5002792.
  17. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
  18. Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 16318-16352.
  19. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/ file/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Paper-Conference.pdf.
  20. D. de Chiusole and L. Stefanutti. Modeling skill dependence in probabilistic competence structures. Electronic Notes in Discrete Mathematics, 42:41-48, 2013. ISSN 1571-0653. doi: https://doi. org/10.1016/j.endm.2013.05.144. URL https://www.sciencedirect.com/science/ article/pii/S1571065313001479. · DOI
  21. Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 2169-2176, 2017. doi: 10.1109/ICRA.2017. 7989250. · DOI
  22. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021.
  23. Harrison Edwards and Amos J. Storkey. Censoring representations with an adversary. CoRR, abs/1511.05897, 2015. URL https://api.semanticscholar.org/CorpusID: 4986726. · arXiv
  24. David Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. CoRR, abs/1312.4314, 2013. URL https://api.semanticscholar. org/CorpusID:11492613. · arXiv
  25. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral ex- planation with amnesic counterfactuals. Transactions of the Association for Computational Lin- guistics, 9:160-175, 2020. URL https://api.semanticscholar.org/CorpusID: 227408471.
  26. Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759, 2023. · arXiv
  27. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Man- deep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1-48, 2021. URL http://jmlr.org/papers/v22/20-1307.html.
  28. Chelsea Finn, Tianhe Yu, Justin Fu, P. Abbeel, and Sergey Levine. Generalizing skills with semi-supervised reinforcement learning. ArXiv, abs/1612.00429, 2016. URL https://api. semanticscholar.org/CorpusID:8685592. · arXiv
  29. Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Ma- chine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1180-1189, Lille, France, 07-09 Jul 2015. PMLR. URL https://proceedings.mlr.press/v37/ ganin15.html.
  30. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural net- works. Journal of Machine Learning Research, 17(59):1-35, 2016. URL http://jmlr. org/papers/v17/15-239.html.
  31. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462, 2020. · arXiv
  32. Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Good- man, and Christopher Potts. Inducing causal fstructure for interpretable neural networks. In International Conference on Machine Learning, 2022a.
  33. Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Good- man, and Christopher Potts. Inducing causal structure for interpretable neural networks. In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 7324-7338. PMLR, 17-23 Jul 2022b. URL https://proceedings.mlr.press/v162/geiger22a.html.
  34. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Processing Systems, 2014. URL https://api.semanticscholar.org/CorpusID: 261560300.
  35. C. A. E. Goodhart. Problems of Monetary Management: The UK Experience. Macmillan Educa- tion UK, London, 1984. ISBN 978-1-349-17295-5. doi: 10.1007/978-1-349-17295-5 4. URL https://doi.org/10.1007/978-1-349-17295-5_4. · DOI
  36. Phillip Huang Guo, Aaquib Syed, Abhay Sheshadri, Aidan Ewart, and Gintare Karolina Dziugaite. Robust unlearning via mechanistic localizations. In ICML 2024 Workshop on Mechanistic Inter- pretability, 2024. URL https://openreview.net/forum?id=06pNzrEjnH.
  37. Suchin Gururangan, Michael Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. Demix layers: Disentangling domains for modular language modeling. In North American Chapter of the Association for Computational Linguistics, 2021. URL https://api.semanticscholar. org/CorpusID:236976189.
  38. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
  39. Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and Transfer of Modulated Locomotor Controllers. arXiv e-prints, art. arXiv:1610.05182, October 2016. doi: 10.48550/arXiv.1610.05182. · DOI
  40. Peter Henderson, Eric Mitchell, Christopher Manning, Dan Jurafsky, and Chelsea Finn. Self- destructing models: Increasing the costs of harmful dual uses of foundation models. In Pro- ceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, AIES '23, pp. 287296, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702310. doi: 10.1145/3600211.3604690. URL https://doi.org/10.1145/3600211.3604690. · DOI
  41. John Hewitt, John Thickstun, Christopher D. Manning, and Percy Liang. Backpack language mod- els. In Proceedings of the Association for Computational Linguistics. Association for Computa- tional Linguistics, 2023.
  42. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- ternational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr. press/v97/houlsby19a.html.
  43. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328-339, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1031. URL https://aclanthology.org/P18-1031. · DOI · Paper page
  44. Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, and Chun ying Huang. Safe lora: the silver lining of reducing safety risks when fine-tuning large language mod- els. ArXiv, abs/2405.16833, 2024. URL https://api.semanticscholar.org/ CorpusID:270063864. · arXiv
  45. Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi, Josh Kimball, and Ling Liu. Antidote: Post- fine-tuning safety alignment for large language models against harmful fine-tuning. arXiv preprint arXiv:2408.09600, 2024. · arXiv
  46. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https://openreview.net/forum?id= 6t0Kwf8-jrj.
  47. Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=FCnohuR6AnM.
  48. Gal Kaplun, Andrey Gurevich, Tal Swisa, Mazor David, Shai Shalev-Shwartz, and eran malach. Less is more: Selective layer finetuning with subtuning, 2024. URL https://openreview. net/forum?id=sOHVDPqoUJ.
  49. Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, and Joar Max Viktor Skalse. Goodhart's law in reinforcement learning. In The Twelfth International Con- ference on Learning Representations, 2024. URL https://openreview.net/forum? id=5o9G4XF1LI.
  50. Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. · arXiv
  51. Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013. URL https://api.semanticscholar.org/CorpusID:216078090. · arXiv
  52. Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Hal Daum III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5338-5348. PMLR, 13-18 Jul 2020. URL https://proceedings. mlr.press/v119/koh20a.html.
  53. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
  54. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mi- halcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint arXiv:2401.01967, 2024. · arXiv
  55. Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=APuPRxjHvZ.
  56. Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. ArXiv, abs/2310.20624, 2023. URL https://api. semanticscholar.org/CorpusID:264808400. · arXiv
  57. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024. · arXiv
  58. Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Chris Liu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. Rethinking machine unlearning for large language models. ArXiv, abs/2402.08787, 2024. URL https://api.semanticscholar.org/CorpusID:267657624. · arXiv
  59. Tyler Lizzo and Larry Heck. Unlearn efficient removal of knowledge in large language models, 2024. URL https://arxiv.org/abs/2408.04140. · arXiv
  60. Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. September 2018. URL https://openreview.net/forum?id=Bkg6RiCqY7.
  61. Jakub Łucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramr, and Javier Rando. An adversarial perspective on machine unlearning for ai safety, 2024. URL https://arxiv. org/abs/2409.18025.
  62. Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight meth- ods to evaluate robust unlearning in llms, 2024. URL https://arxiv.org/abs/2402. 16835.
  63. Pattie Maes and Rodney A Brooks. Learning to coordinate behaviors. In AAAI, volume 90, pp. 796-802. Boston, MA, 1990.
  64. Sridhar Mahadevan and Jonathan Connell. Automatic programming of behavior-based robots using reinforcement learning. Artificial Intelligence, 55(2):311-365, 1992. ISSN 0004-3702. doi: https: //doi.org/10.1016/0004-3702(92)90058-6. URL https://www.sciencedirect.com/ science/article/pii/0004370292900586. · DOI
  65. Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
  66. Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to mul- tiple tasks by learning to mask weights. In Proceedings of the European conference on computer vision (ECCV), pp. 67-82, 2018.
  67. Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4402-4412. PMLR, 09-15 Jun 2019. URL https://proceedings. mlr.press/v97/mathieu19a.html.
  68. Tom McGrath, Matthew Rahtz, János Kramár, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. ArXiv, abs/2307.15771, 2023. URL https://api.semanticscholar.org/CorpusID:260334719. · arXiv
  69. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. Advances in Neural Information Processing Systems, 36, 2022. arXiv:2202.05262. · arXiv
  70. Joseph Miller, Bilal Chughtai, and William Saunders. Transformer circuit evaluation metrics are not robust. In First Conference on Language Modeling, 2024. URL https://openreview. net/forum?id=zSf8PJyQb2.
  71. Amirkeivan Mohtashami, Martin Jaggi, and Sebastian U Stich. Masked training of neural networks with partial gradients. In Proceedings of the 25th International Conference on Artificial Intelli- gence and Statistics, 2022.
  72. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024-001, 2020.
  73. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022. · arXiv
  74. Stephen M. Omohundro. The basic ai drives. In Proceedings of the 2008 Conference on Artificial General Intelligence 2008: Proceedings of the First AGI Conference, pp. 483492, NLD, 2008. IOS Press. ISBN 9781586038335.
  75. Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mit- tal. Lottery ticket adaptation: Mitigating destructive interference in LLMs. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Op- timization (WANT@ICML 2024), 2024a. URL https://openreview.net/forum?id= qD2eFNvtw4.
  76. Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mittal. Lottery ticket adaptation: Mitigating destructive interference in LLMs, 2024b. URL http: //arxiv.org/abs/2406.16797.
  77. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
  78. Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from llms? ob- jectives for defending against extraction attacks. ArXiv, abs/2309.17410, 2023. URL https: //api.semanticscholar.org/CorpusID:263311025. · arXiv
  79. Guilherme Penedo, Hynek Kydlek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. (arXiv:2406.17557), 2024. doi: 10.48550/arXiv.2406.17557. URL http://arxiv.org/abs/2406.17557. · DOI · Paper page
  80. Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. Mad-x: An adapter-based frame- work for multi-task cross-lingual transfer. In Conference on Empirical Methods in Natural Language Processing, 2020. URL https://api.semanticscholar.org/CorpusID: 218470133.
  81. Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin O. Steitz, Stefan Roth, Ivan Vulic, and Iryna Gurevych. xgqa: Cross-lingual visual question answering. In Findings, 2021. URL https://api.semanticscholar.org/CorpusID:237490295.
  82. Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. Lifting the curse of multilinguality by pre-training modular transformers. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3479-3495, Seattle, United States, July 2022. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.255. URL https: //aclanthology.org/2022.naacl-main.255. · DOI
  83. Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Ponti. Modular deep learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/ forum?id=z9EkXfvxta. Survey Certification.
  84. Nicholas Pochinkov and Nandi Schoots. Dissecting language models: Machine unlearning via selective pruning, 2024. URL https://arxiv.org/abs/2403.01267. · arXiv
  85. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http: //jmlr.org/papers/v21/20-074.html.
  86. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Annual Meeting of the As- sociation for Computational Linguistics, 2020. URL https://api.semanticscholar. org/CorpusID:215786522.
  87. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30.
  88. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_ files/paper/2017/file/e7b24b112a44fdd9ee93bdf998c6ca0e-Paper.pdf.
  89. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi- domain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  90. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama 2 via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15504-15522, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024. acl-long.828.
  91. Amir Rosenfeld and John K. Tsotsos. Intriguing properties of randomly weighted networks: Gen- eralizing while learning next to nothing. 2019 16th Conference on Computer and Robot Vi- sion (CRV), pp. 9-16, 2018. URL https://api.semanticscholar.org/CorpusID: 3657091.
  92. Amir Rosenfeld and John K. Tsotsos. Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing. In 2019 16th Conference on Computer and Robot Vision (CRV), pp. 9-16, May 2019. doi: 10.1109/CRV.2019.00010. URL https: //ieeexplore.ieee.org/document/8781620. · DOI
  93. Jerome H Saltzer and Michael D Schroeder. The protection of information in computer systems. Proceedings of the IEEE, 63(9):1278-1308, 1975.
  94. Pierangela Samarati and Sabrina Capitani de Vimercati. Access control: Policies, models, and mechanisms. In Riccardo Focardi and Roberto Gorrieri (eds.), Foundations of Security Analysis and Design, pp. 137-196, Berlin, Heidelberg, 2001. Springer Berlin Heidelberg. ISBN 978-3- 540-45608-7.
  95. R.S. Sandhu and P. Samarati. Access control: principle and practice. IEEE Communications Mag- azine, 32(9):40-48, 1994. doi: 10.1109/35.312842. · DOI
  96. Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural Computation, 4:863-879, 1992. URL https://api.semanticscholar.org/CorpusID:2142508.
  97. Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, et al. Targeted latent ad- versarial training improves robustness to persistent harmful behaviors in llms. arXiv preprint arXiv:2407.15549, 2024. · arXiv
  98. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview.net/forum?id=zWqr3MQuNs.
  99. Irene Solaiman and Christy Dennison. Process for adapting language models to society (palms) with values-targeted datasets. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 5861-5873.
  100. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_ files/paper/2021/file/2e855f9489df0712b4bd8ea9e2848c5a-Paper.pdf.
  101. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
  102. Xu Sun, Xuancheng Ren, Shuming Ma, and Houfeng Wang. meprop: Sparsified back propagation for accelerated deep learning with reduced overfitting. In International Conference on Machine Learning, pp. 3299-3308. PMLR, 2017a.
  103. Xu Sun, Xuancheng Ren, Shuming Ma, and Houfeng Wang. meProp: Sparsified back propagation for accelerated deep learning with reduced overfitting. In Proceedings of the 34 th International Conference on Machine Learning, 2017b.
  104. Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. ArXiv, abs/2111.09839, 2021. URL https://api.semanticscholar.org/ CorpusID:244345839. · arXiv
  105. Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http://incompleteideas.net/book/the-book-2nd. html.
  106. Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, Andy Zou, Dawn Song, Bo Li, Dan Hendrycks, and Mantas Mazeika. Tamper-resistant safeguards for open-weight llms, 2024. URL https: //arxiv.org/abs/2408.00761.
  107. Alex Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal policies tend to seek power. Advances in Neural Information Processing Systems, 34:23063-23074, 2021.
  108. Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDi- armid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248, 2023. · arXiv
  109. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, L. Wang, An- tonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. ArXiv, abs/2211.14275, 2022. URL https://api. semanticscholar.org/CorpusID:254017497. · arXiv
  110. Fabio Urbina, Filippa Lentzos, Cdric Invernizzi, and Sean Ekins. Dual use of artificial-intelligence- powered drug discovery. Nature Machine Intelligence, 4(3):189-191, March 2022. ISSN 2522- 5839. doi: 10.1038/s42256-022-00465-9. URL https://www.nature.com/articles/ s42256-022-00465-9. Publisher: Nature Publishing Group. · DOI · Paper page
  111. A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
  112. A. Waibel and J. Hampshire II. The meta-pi network: Building distributed knowledge represen- tations for robust multisource pattern recognition. IEEE Transactions on Pattern Analysis & Machine Intelligence, 14(07):751-769, jul 1992. ISSN 1939-3539. doi: 10.1109/34.142911. · DOI
  113. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https: //openreview.net/forum?id=NpsVSN6o4ul.
  114. Xin Wang, Hong Chen, Si'ao Tang, Zihao Wu, and Wenwu Zhu. Disentangled representation learn- ing. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-20, 2024. doi: 10.1109/TPAMI.2024.3420937. · DOI
  115. Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hen- dricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detox- ifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2447-2469, 2021.
  116. Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
  117. Rui Xin, Chudi Zhong, Zhi Chen, Takuya Takagi, Margo I. Seltzer, and Cynthia Rudin. Ex- ploring the whole rashomon set of sparse decision trees. Advances in neural information pro- cessing systems, 35:14071-14084, 2022. URL https://api.semanticscholar.org/ CorpusID:252355323.
  118. Xin Yi, Shunfan Zheng, Linlin Wang, Xiaoling Wang, and Liang He. A safety realignment frame- work via subspace-oriented model fusion for large language models. ArXiv, abs/2405.09055, 2024. URL https://api.semanticscholar.org/CorpusID:269773206. · arXiv
  119. Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv: Computer Vision and Pattern Recognition, 2017. URL https://api.semanticscholar. org/CorpusID:46294020.
  120. Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization, October 2019. URL http://arxiv.org/abs/1910.07467. arXiv:1910.07467 [cs, stat]. · arXiv
  121. Enyan Zhang, Michael A. Lepori, and Ellie Pavlick. Instilling inductive biases with subnetworks, 2024. URL https://openreview.net/forum?id=B4nhr6OJWI.
  122. Haojie Zhang, Ge Li, Jia Li, Zhongjin Zhang, Yuqi Zhu, and Zhi Jin. Fine-tuning pre-trained lan- guage models effectively by optimizing subnetworks adaptively. Advances in Neural Information Processing Systems, 35:21442-21454, 2022.
  123. Jinghan Zhang, shiqi chen, Junteng Liu, and Junxian He. Composing parameter- efficient modules with arithmetic operation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 12589-12610.
  124. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 299a08ee712d4752c890938da99a77c6-Paper-Conference.pdf.

Cited by

Sorted by citations

Metadata

Authors
Cloud, A., Goldman-Wetzler, J., Wybitul, E., Miller, J., & Turner, A. M.
Venue
arXiv
Publisher
arXiv
Published
2024
DOI
10.48550/arxiv.2410.04332
arXiv
2410.04332
References
4
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review