CoChat
Sign Up
PreprintCited by 45Cryptography and Data SecurityPrivacy-Preserving Technologies in DataNeural Networks and Applications

The DOI matches its DataCite record.

LEACE: Perfect linear concept erasure in closed form

Open access
Belrose, Nora, Schneider-Joseph, David, Ravfogel, Shauli, Cotterell, Ryan, Raff, Edward, Biderman, Stella·2023·arXiv
DOI 10.48550/arxiv.2306.03819arXiv 2306.03819
Open this paper in CoChat — free accountOpen PDF

Abstract

Concept erasure aims to remove specified features from an embedding. It can improve fairness (e.g. preventing a classifier from using gender or race) and interpretability (e.g. removing a concept to observe changes in model behavior). We introduce LEAst-squares Concept Erasure (LEACE), a closed-form method which provably prevents all linear classifiers from detecting a concept while changing the embedding as little as possible, as measured by a broad class of norms. We apply LEACE to large language models with a novel procedure called "concept scrubbing," which erases target concept information from every layer in the network. We demonstrate our method on two tasks: measuring the reliance of language models on part-of-speech information, and reducing gender bias in BERT embeddings. Code is available at https://github.com/EleutherAI/concept-erasure.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.03819

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Works addressing bias in LLMs commonly rely on bias-sensitive prediction probability.

    “Works addressing bias in LLMs ( e.g. , gender, race or stereotype) (Belrose et al 2023 ; Limisiewicz and Mareček 2022; Yu et al 2023) commonly rely on the bias-sensitive prediction probability.”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Belrose et al 2023

    In the profession-prediction evaluation section (page unavailable), Belrose et al. measure gender-related classifier bias, so the paper is an example of work addressing bias. But it says its bias measure is the gap in true positive rates between protected groups, not bias-sensitive prediction probability; it also does not establish what such works commonly rely on. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • A limited number of authors employ datasets with tasks that are closely aligned with the knowledge intended for forgetting.

    “In contrast, a limited number of authors employ datasets with tasks that are closely aligned with the knowledge intended for forgetting (Limisiewicz and Mareček 2022; Belrose et al 2023; Pochinkov and Schoots 2023; Ni et al 2023).”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Belrose et al 2023

  • Belrose et al (2023) and Wu et al (2023) use the forgetting dataset to measure capabilities in a task disjoint from that to be forgotten.

    “Same dataset but different task . The forgetting dataset is used to measure the model's capabilities in a task disjoint of that to be forgotten (Belrose et al 2023; Wu et al 2023).”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Belrose et al 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 40

  1. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373, 2023. · arXiv
  2. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. Man is to computer programmer as woman is to homemaker? De- biasing word embeddings. Advances in Neural Information Processing Systems, 29:4349-4357, 2016. URL https://proceedings.neurips.cc/paper/2016/file/ a486cd07e4ac3d270571622f4f316ec5-Paper.pdf.
  3. Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. Adversarial deep averaging networks for cross-lingual sentiment classification. Transactions of the Associa- tion for Computational Linguistics, 6:557-570, 2018. URL https://aclanthology.org/ Q18-1039.
  4. Verna Dankers, Christopher Lucas, and Ivan Titov. Can transformer be too compositional? Analysing idiom processing in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3608-3626, 2022.
  5. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexan- dra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* '19, page 120-128, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361255. doi: 10.1145/3287560.3287572. URL https://doi.org/10.1145/3287560.3287572. · DOI · Paper page
  6. Sunipa Dev, Tao Li, Jeff M. Phillips, and Vivek Srikumar. OSCaR: Orthogonal subspace correction and rectification of biases in word embeddings. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 5034-5050, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Lin- guistics. doi: 10.18653/v1/2021.emnlp-main.411. URL https://aclanthology.org/2021. emnlp-main.411. · DOI
  7. Harrison Edwards and Amos Storkey. Censoring representations with an adversary. In In- ternational Conference in Learning Representations, pages 1-14, May 2016. URL https: //arxiv.org/abs/1511.05897.
  8. Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11-21, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1002. URL https://aclanthology.org/D18-1002. · DOI
  9. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175, 2021. doi: 10.1162/tacl_a_00359. URL https://aclanthology. org/2021.tacl-1.10. · DOI · Paper page
  10. Thomas S. Ferguson. Mathematical Statistics. Academic Press, Cambridge, MA, 1967.
  11. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. · arXiv
  12. Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. In International Conference on Machine Learning, pages 7324-7338. PMLR, 2022.
  13. Christopher Grimsley, Elijah Mayfield, and Julia R.S. Bursten. Why attention is not explanation: Surgical intervention and causal reasoning about neural models. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1780-1790, Marseille, France, May 2020. European Language Resources Association. URL https://aclanthology.org/2020. lrec-1.220.
  14. Pantea Haghighatkhah, Wouter Meulemans, Bettina Speckmann, Jérôme Urhausen, and Kevin Verbeek. Obstructing classification via projection. In Filippo Bonchi and Simon J. Puglisi, editors, 46th International Symposium on Mathematical Foundations of Computer Science, Leibniz International Proceedings in Informatics, LIPIcs. Schloss Dagstuhl -Leibniz-Zentrum für Informatik, 2021.
  15. Pantea Haghighatkhah, Antske Fokkens, Pia Sommerauer, Bettina Speckmann, and Kevin Verbeek. Better hit the nail on the head than beat around the bush: Removing protected attributes with a single projection. pages 8395-8416, December 2022. doi: 10.18653/v1/2022. emnlp-main.575. URL https://aclanthology.org/2022.emnlp-main.575. · DOI
  16. Evan Hernandez and Jacob Andreas. The low-dimensional linear geometry of contextualized word representations. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 82-93, 2021.
  17. Matthäus Kleindessner, Michele Donini, Chris Russell, and Muhammad Bilal Zafar. Efficient fair PCA for fair representation learning. In International Conference on Artificial Intelligence and Statistics, pages 5250-5270. PMLR, 2023.
  18. Abhinav Kumar, Chenhao Tan, and Amit Sharma. Probing classifiers are unreliable for concept removal and detection. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 17994-18008.
  19. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/ paper/2022/file/725f5e8036cc08adeba4a7c3bcbc6f2c-Paper-Conference.pdf.
  20. Matt J. Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in Neural Information Processing Systems, 30, 2017.
  21. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, et al. Universal dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 92-97, 2013.
  22. Vassilina Nikoulina, Maxat Tezekbayev, Nuradil Kozhakhmet, Madina Babazhanova, Matthias Gallé, and Zhenisbek Assylbekov. The rediscovery hypothesis: Language models need to meet linguistics. Journal of Artificial Intelligence Research, 72:1343-1384, 2021.
  23. Hamed Nilforoshan, Johann D. Gaebler, Ravi Shroff, and Sharad Goel. Causal conceptions of fairness and their consequences. In International Conference on Machine Learning, pages 16848-16887. PMLR, 2022.
  24. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajic, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4034-4043, 2020.
  25. Judea Pearl. Causality. Cambridge University Press, Cambridge, UK, 2 edition, 2009. ISBN 978-0-521-89560-6. doi: 10.1017/CBO9780511803161. · DOI · Paper page
  26. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.647. URL https://aclanthology.org/2020.acl-main.647. · DOI · Paper page
  27. Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. Counterfactual interventions reveal the causal effect of relative clause representations on agreement prediction. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 194-209, 2021.
  28. Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept erasure. In International Conference on Machine Learning, pages 18400-18421. PMLR, 2022.
  29. Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Adversarial concept erasure in kernel space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6034-6055, 2022.
  30. Shauli Ravfogel, Yoav Goldberg, and Ryan Cotterell. Log-linear guardedness and its im- plications. In Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 9413-9431, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.523. URL https://aclanthology.org/2023.acl-long.523. · DOI
  31. Bashir Sadeghi and Vishnu Boddeti. On the fundamental trade-offs in learning invariant representations. arXiv preprint arXiv:2109.03386, 2021. URL https://openreview.net/ pdf?id=KOk7mUGspN9. · arXiv
  32. Bashir Sadeghi, Runyi Yu, and Vishnu Boddeti. On the global optima of kernelized adversarial representation learning. In 2019 IEEE/CVF International Conference on Computer Vision, pages 7970-7978. IEEE, 2019. URL http://hal.cse.msu.edu/assets/pdfs/papers/ 2019-iccv-kernel-adversarial-representation-learning.pdf.
  33. Shun Shao, Yftah Ziser, and Shay B. Cohen. Gold doesn't always glitter: Spectral removal of linear and nonlinear guarded attribute information. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1611-1622, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. URL https: //aclanthology.org/2023.eacl-main.118.
  34. Shun Shao, Yftah Ziser, and Shay B. Cohen. Erasure of unaligned attributes from neural representations. Transactions of the Association for Computational Linguistics, 11:488-510, 2023. doi: 10.1162/tacl_a_00558. URL https://aclanthology.org/2023.tacl-1.29. · DOI
  35. Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, 2019.
  36. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models, 2023. URL http://arxiv.org/abs/2302.13971. · arXiv
  37. James Francis Woodward. Making Things Happen: A Theory of Causal Explanation explanation. Oxford University Press, 2005.
  38. Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. In Advances in Neural Information Processing Systems, volume 30, pages 585-596, 2017. URL https://dl.acm.org/doi/10.5555/3294771. 3294827. · DOI
  39. Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. In 8th International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=r1eBeyHFDH.
  40. Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, page 335-340, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450360128. URL https://doi.org/10.1145/3278721.3278779. · DOI · Paper page

Cited by

Sorted by citations

Related

Metadata

Authors
Belrose, N., Schneider-Joseph, D., Ravfogel, S., Cotterell, R., Raff, E., & Biderman, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
66044-66063
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2306.03819
arXiv
2306.03819
OpenAlex
W7133241667
Cited by
44
References
0
Citations by year
2026: 28 · 2025: 6 · 2024: 5 · 2023: 5
FWCI
7.81
Type
conference paper
Language
en
Open access
Licence
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Cryptography and Data Security
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
concept erasure, analytical solution, linear algebra

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review