CoChat
Sign Up
PreprintCited by 295Explainable Artificial Intelligence (XAI)Topic ModelingAdversarial Robustness in Machine Learning

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models

Open access
Marks, Samuel, Rager, Can, Michaud, Eric J., Belinkov, Yonatan, Bau, David, Mueller, Aaron·2024·arXiv
DOI 10.48550/arxiv.2403.19647arXiv 2403.19647
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce methods for discovering and applying sparse feature circuits. These are causally implicated subnetworks of human-interpretable features for explaining language model behaviors. Circuits identified in prior work consist of polysemantic and difficult-to-interpret units like attention heads or neurons, rendering them unsuitable for many downstream applications. In contrast, sparse feature circuits enable detailed understanding of unanticipated mechanisms. Because they are based on fine-grained units, sparse feature circuits are useful for downstream tasks: We introduce SHIFT, where we improve the generalization of a classifier by ablating features that a human judges to be task-irrelevant. Finally, we demonstrate an entirely unsupervised and scalable interpretability pipeline by discovering thousands of sparse feature circuits for automatically discovered model behaviors.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.19647

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Error nodes contain 'everything else', making them an inadequate solution to the reconstruction error problem.

    “However, while the sparse autoencoders identified interpretable latents, the error nodes contain 'everything else', making them an inadequate solution to the problem.”

    From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Marks et al., 2024

    Section 2 (Formulation; page unavailable) defines SAE error terms as components not captured by interpretable features, and Section 8 (Limitations; page unavailable) says components not captured by SAEs remain uninterpretable. That supports the concern behind 'everything else,' but the paper describes incorporating error nodes as a principled way to account for them; it does not call them an inadequate solution to reconstruction error. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Faster alternatives that work well in practice include gradient-based approximations to activation patching, such as attribution patching, AtP*, and integrated gradients.

    “Faster alternatives that work well in practice (Marks et al., 2024;Templeton et al., 2024) include gradient-based approximations to activation patching, such as attribution patching (Nanda, 2023a;Syed et al., 2024), AtP* (Kramár et al., 2024), and integrated gradients (Sundararajan et al., 2017).”

    From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Marks et al., 2024

    Section 2 (page not available) says attribution patching approximates activation patching for many components in parallel using two forward passes and one backward pass. It also describes an integrated-gradients approximation as more accurate but more expensive. The passages do not identify AtP* as a method used or evaluated by this paper, so the full list is not supported by Marks et al. alone. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Prior circuit discovery work has attempted to decompose models in either architectural bases or sparse autoencoder latents, which are imperfect ways to decompose neural networks for mechanistic interpretability.

    “Perhaps most importantly, prior circuit discovery work has attempted to decompose models in either architectural bases (Wang et al., 2023;Conmy et al., 2023) or sparse autoencoder latents (Marks et al., 2024;Huben et al., 2024), which are imperfect ways to decompose neural networks for mechanistic interpretability (Section 2.1.2).”

    From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Marks et al., 2024

  • More recent work has used SDL latents for nodes.

    “However, more recent work has also used SDL latents for nodes (Marks et al., 2024)”

    From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Marks et al., 2024

  • Attribution patching was designed to identify potential relevant candidates for further testing through intervention.

    “Attribution patching Syed et al. (2024) was designed to identify potential relevant candidates for further testing through intervention, which becomes more important as the number of components expands significantly through sparse dictionary learning (Marks et al., 2024).”

    From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Marks et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 69

  1. Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, 2022.
  2. Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023, LEACE: Perfect linear concept erasure in closed form.
  3. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
  4. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, Weak-to- Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. Computing Re- search Repository, arXiv:2312.09390, 2023. · arXiv
  5. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback, Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. Survey Certification.
  6. Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. Sud- den drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. In The Twelfth International Conference on Learning Representations, 2024, Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs.
  7. Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. In- fogan: interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, pp. 2180-2188, Red Hook, NY, USA, 2016.
  8. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Thirty- seventh Conference on Neural Information Processing Systems, 2023, Towards Automated Circuit Discovery for Mechanistic Interpretability.
  9. Elliot Creager, Joern-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2189-2200. PMLR, 18-24 Jul 2021, Environment Inference for Invariant Learning.
  10. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* '19, pp. 120-128, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361255, Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting. · DOI · Paper page
  11. Guillaume Desjardins, Aaron Courville, and Yoshua Bengio. Disentangling factors of variation via generative entangling. Computing Research Repository, arXiv:1210.5474, 2012. · arXiv
  12. Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1828-1843, Online, August 2021. Association for Computational Linguistics, Causal Analysis of Syntactic Agree- ment Mechanisms in Neural Language Models.
  13. Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting CLIP's image representation via text-based decomposition. Computing Research Repository, arXiv:2310.05916, 2024. · arXiv
  14. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling. Computing Research Repository, arXiv:2101.00027, 2020. · arXiv
  15. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. Computing Research Repository, arXiv:2406.04093, 2024. · arXiv
  16. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 9574-9586.
  17. Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Good- man, and Christopher Potts. Inducing causal structure for interpretable neural networks. In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 7324-7338. PMLR, 17-23 Jul 2022, Inducing Causal Structure for Interpretable Neural Networks.
  18. Atticus Geiger, Chris Potts, and Thomas Icard. Causal abstraction for faithful model interpretation. Computing Research Repository, arXiv:2301.04709, 2023. · arXiv
  19. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pp. 12216-12235, Singapore, December 2023. Association for Computational Linguistics, Dissecting Recall of Factual Associations in Auto-Regressive Language Models.
  20. Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, inter- pretable attention heads in the wild. Computing Research Repository, arXiv:2312.09230, 2023. · arXiv
  21. Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023, How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model.
  22. Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In ICML 2024 Workshop on Mechanistic In- terpretability, 2024, Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms.
  23. Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe. The unreasonable effectiveness of easy training data for hard tasks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7002-7024, Bangkok, Thailand, August 2024. Association for Computational Linguistics, The Unreasonable Effectiveness of Easy Training Data for Hard Tasks.
  24. T. He, Z. Li, Y. Gong, Y. Yao, X. Nie, and Y. Yin. Exploring linear feature disentanglement for neural networks. In 2022 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6, Los Alamitos, CA, USA, jul 2022. IEEE Computer Society, Exploring Linear Feature Dis- entanglement for Neural Networks. · DOI
  25. Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017, beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.
  26. Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-accuracy. In Bernhard Schölkopf, Caroline Uhler, and Kun Zhang (eds.), Proceedings of the First Conference on Causal Learning and Reasoning, volume 177 of Proceedings of Machine Learning Research, pp. 336-351. PMLR, 11-13 Apr 2022, Simple data balancing achieves competitive worst-group-accuracy.
  27. Shadi Iskander, Kira Radinsky, and Yonatan Belinkov. Shielded representations: Protecting sensitive attributes through iterative gradient-based projection. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 5961-5977, Toronto, Canada, July 2023. Association for Computational Linguistics, Shielded Representations: Protecting Sensitive Attributes Through Iterative Gradient-Based Projection.
  28. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Proceedings of the 35th International Conference on Machine Learning, pp. 2668- 2677. PMLR, 2018.
  29. Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Pro- ceedings of Machine Learning Research, pp. 2649-2658. PMLR, 10-15 Jul 2018, Disentangling by Factorising.
  30. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, Adam: A Method for Stochastic Optimization. CoRR, abs/1412.6980, 2014. · arXiv
  31. Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is suffi- cient for robustness to spurious correlations. Computing Research Repository, arXiv:2204.02937, 2023. · arXiv
  32. János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. AtP*: An efficient and scalable method for localizing llm behaviour to components. Computing Research Repository, arXiv:2403.00745, 2024. · arXiv
  33. Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6781-6792. PMLR, 18-24 Jul 2021, Just Train Twice: Improving Group Robustness without Training Group Information.
  34. Alireza Makhzani and Brendan J. Frey. k-sparse autoencoders, k-Sparse Autoencoders. Computing Research Repository, abs/1312.5663, 2013. · arXiv
  35. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. Advances in Neural Information Processing Systems, 36, 2022. arXiv:2202.05262. · arXiv
  36. Eric J Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023, The Quantization Model of Neural Scaling.
  37. Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: Training debiased classifier from biased classifier. In Proceedings of the 34th International Con- ference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020.
  38. Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective. Computing Research Repository, arXiv:2209.00626, 2024. · arXiv
  39. Tuomas Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bot- tleneck models. In The Eleventh International Conference on Learning Representations, 2023, Label-free Concept Bottleneck Models.
  40. Yonatan Oren, Shiori Sagawa, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust lan- guage modeling. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4227-4237, Hong Kong, China, November 2019. Association for Computational Linguistics, Distributionally Robust Language Modeling.
  41. Hadas Orgad and Yonatan Belinkov. BLIND: Bias removal with no demographics. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8801-8821, Toronto, Canada, July 2023. Association for Computational Linguistics, BLIND: Bias Removal With No Demographics.
  42. Judea Pearl. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncer- tainty in Artificial Intelligence, UAI'01, pp. 411-420, San Francisco, CA, USA, 2001.
  43. Morgan Kaufmann Publishers Inc. ISBN 1558608001.
  44. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
  45. William Peebles, John Peebles, Jun-Yan Zhu, Alexei A. Efros, and Antonio Torralba. The hessian penalty: A weak prior for unsupervised disentanglement. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
  46. Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In Proceedings of the 2024 International Conference on Learning Representations, 2024. arXiv:2402.14811. · arXiv
  47. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoen- coders. Computing Research Repository, arXiv:2404.16014, 2024a. · arXiv
  48. Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Au- toencoders. Computing Research Repository, arXiv:2407.14435, 2024b. · arXiv
  49. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guard- ing protected attributes by iterative nullspace projection. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7237-7256, Online, July 2020. Association for Computational Linguistics, Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection.
  50. Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept erasure. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18400-18421. PMLR, 17-23 Jul 2022a, Linear Adversarial Concept Erasure.
  51. Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Adversarial concept erasure in kernel space. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6034-6055, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics, Adversarial Concept Erasure in Kernel Space.
  52. James M. Robins and Sander Greenland. Identifiability and exchangeability for direct and indirect effects, Identifiability and Exchangeability for Direct and Indirect Effects. Epidemiology, 3(2): 143-155, 1992. ISSN 10443983.
  53. Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020, Distribution- ally Robust Neural Networks.
  54. Jürgen Schmidhuber. Learning Factorial Codes by Predictability Minimization, Learning Factorial Codes by Predictability Minimization. Neural Computation, 4(6):863-879, 11 1992. ISSN 0899- 7667. · DOI
  55. Johannes Schneider and Michalis Vlachos. Explaining neural networks by decoding layer activa- tions. In Advances in Intelligent Data Analysis XIX: 19th International Symposium on Intelligent Data Analysis, IDA 2021, Porto, Portugal, April 26-28, 2021, Proceedings, pp. 63-75, Berlin, Heidelberg, 2021. Springer-Verlag. ISBN 978-3-030-74250-8, Explaining Neural Networks by Decoding Layer Activations. · DOI
  56. Nimit Sharad Sohoni, Maziar Sanjabi, Nicolas Ballas, Aditya Grover, Shaoliang Nie, Hamed Firooz, and Christopher Re. BARACK: Partially supervised group robustness with guarantees. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022, BARACK: Partially Supervised Group Robustness With Guarantees.
  57. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning -Volume 70, ICML'17, pp. 3319-3328. JMLR.org, 2017.
  58. Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated cir- cuit discovery. In NeurIPS Workshop on Attributing Model Behavior at Scale, 2023, Attribution Patching Outperforms Automated Circuit Discovery.
  59. Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. In Proceedings of the 2024 International Conference on Learning Representations, 2024.
  60. Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Towards debiasing NLU models from unknown biases. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7597-7610, Online, November 2020. Association for Computational Linguistics, Towards Debiasing NLU Models from Unknown Biases.
  61. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12388-12401.
  62. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023, Interpretability in the Wild: a Cir- cuit for Indirect Object Identification in GPT-2 Small.
  63. Tianlu Wang, Xi Victoria Lin, Nazneen Fatema Rajani, Bryan McCann, Vicente Ordonez, and Caim- ing Xiong. Double-hard debias: Tailoring word embeddings for gender bias mitigation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5443-5453, Online, July 2020. As- sociation for Computational Linguistics, Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigation.
  64. Yadollah Yaghoobzadeh, Soroush Mehri, Remi Tachet des Combes, T. J. Hazen, and Alessandro Sordoni. Increasing robustness to spurious correlations using forgettable examples. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 3319- 3332, Online, April 2021. Association for Computational Linguistics, Increasing Robustness to Spurious Correlations using Forgettable Examples.
  65. An Yan, Yu Wang, Yiwu Zhong, Zexue He, Petros Karypis, Zihan Wang, Chengyu Dong, Amil- care Gentili, Chun-Nan Hsu, Jingbo Shang, and Julian McAuley. Robust and interpretable medical image classifiers via concept bottleneck models. Computing Research Repository, arXiv:2310.03182, 2023. · arXiv
  66. Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in lan- guage models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9924-9959, Singapore, December 2023. Association for Computational Linguistics, Characterizing Mechanisms for Fac- tual Recall in Language Models.
  67. John R. Zech, Marcus A. Badgeley, Manway Liu, Anthony B. Costa, Joseph J. Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study, Variable generalization performance of a deep learn- ing model to detect pneumonia in chest radiographs: A cross-sectional study. PLOS Medicine, 15(11):e1002683, November 2018. ISSN 1549-1676. · DOI · Paper page
  68. Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Re. Correct-N- Contrast: A contrastive approach for improving robustness to spurious correlations. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Pro- ceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 26484-26516. PMLR, 17-23 Jul 2022, Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations.
  69. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to AI transparency. Computing Research Repository, arXiv:2310.01405, 2023. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., & Mueller, A.
Affiliations
  • Northeastern University (US)
  • Massachusetts Institute of Technology (US)
  • Technion – Israel Institute of Technology (IL)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2403.19647
arXiv
2403.19647
OpenAlex
W4393336283
Cited by
295
References
1
Citations by year
2026: 119 · 2025: 120 · 2024: 55
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (2141064)
  • Open Philanthropy Project
  • Azrieli Foundation
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
language model interpretability, feature ablation, classifier generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review