CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record, with a small difference in title or year; 2 of the 2 references it cites were found in a registry or an index.

The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis

Mueller, Aaron, Brinkmann, Jannik, Li, Millicent, Marks, Samuel, Pal, Koyena, Prakash, Nikhil, Rager, Can, Sankaranarayanan, Aruna, Sharma, Arnab Sen, Sun, Jiuding, Todd, Eric, Bau, David, Belinkov, Yonatan·2024·arXiv
DOI 10.48550/arxiv.2408.01416arXiv 2408.01416
Open this paper in CoChat — free account

Abstract

Interpretability provides a toolset for understanding how and why neural networks behave in certain ways. However, there is little unity in the field: most studies employ ad-hoc evaluations and do not share theoretical foundations, making it difficult to measure progress and compare the pros and cons of different techniques. Furthermore, while mechanistic understanding is frequently discussed, the basic causal units underlying these mechanisms are often not explicitly defined. In this article, we propose a perspective on interpretability research grounded in causal mediation analysis. Specifically, we describe the history and current state of interpretability taxonomized according to the types of causal units (mediators) employed, as well as methods used to search over mediators. We discuss the pros and cons of each mediator, providing insights as to when particular kinds of mediators and search methods are most appropriate. We argue that this framing yields a more cohesive narrative of the field and helps researchers select appropriate methods based on their research objective. Our analysis yields actionable recommendations for future work, including the discovery of new mediators and the development of standardized evaluations tailored to these goals.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2408.01416

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Figures

Image 1
CC-BY-NC-ND · From the PDF
Image 2
CC-BY-NC-ND · From the PDF
Image 3
CC-BY-NC-ND · From the PDF
Image 4
CC-BY-NC-ND · From the PDF
Image 5
CC-BY-NC-ND · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Scaling Laws for Neural Language Models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×

    “At the same time, however, researchers began to realize the significant performance improvements that could be gained by massively increasing the number of parameters and training corpus sizes of neural networks (Brown et al. 2020; Kaplan et al. 2020).”

  • Anchors: High-Precision Model-Agnostic Explanations

    Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 2×

    “…supervised auxiliary (correlational) models to explain particular model predictions, such as LIME (Ribeiro, Singh, and Guestrin 2016b,a), Anchors (Ribeiro, Singh, and Guestrin 2018), and extensions like CLEAR that explicitly integrate notions of counterfactual fidelity to the output explanations…”

Research works printed but not linked: 190

  1. Amini, Afra, Tiago Pimentel, Clara Meister, and Ryan Cotterell. 2023. Naturalistic causal probing for morpho-syntax. Transactions of the Association for Computational Linguistics, 11:384-403.
  2. Antverg, Omer and Yonatan Belinkov. 2022. On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations.
  3. Arora, Aryaman, Dan Jurafsky, and Christopher Potts. 2024. CausalGym: Benchmarking causal interpretability methods on linguistic tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14638-14663, Association for Computational Linguistics, Bangkok, Thailand.
  4. Arora, Sanjeev, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. A latent variable model approach to PMI-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399.
  5. Arora, Sanjeev, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495.
  6. Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations.
  7. Bastings, Jasmijn, Wilker Aziz, and Ivan Titov. 2019. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2963-2977, Association for Computational Linguistics, Florence, Italy.
  8. Bau, Anthony, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James R. Glass. 2019a. Identifying and controlling important neurons in neural machine translation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net.
  9. Bau, David, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. 2020. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 117(48):30071-30078.
  10. Bau, David, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. 2019b. Visualizing and understanding generative adversarial networks. In International Conference on Learning Representations.
  11. Belinkov, Yonatan. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219.
  12. Belinkov, Yonatan, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861-872, Association for Computational Linguistics, Vancouver, Canada.
  13. Belinkov, Yonatan and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72.
  14. Belrose, Nora, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. LEACE: Perfect linear concept erasure in closed form. In Thirty-seventh Conference on Neural Information Processing Systems.
  15. Bengio, Yoshua, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798-1828.
  16. Bereska, Leonard and Stratis Gavves. 2024. Mechanistic interpretability for AI safety -a review. Transactions on Machine Learning Research. Survey Certification, Expert Certification.
  17. Bills, Steven, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Language models can explain neurons in language models. OpenAI Blog.
  18. Boz, Olcay. 2002. Extracting decision trees from trained neural networks. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '02, page 456-461, Association for Computing Machinery, New York, NY, USA.
  19. Braun, Dan, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey. 2024. Identifying functionally important features with end-to-end sparse dictionary learning. arXiv preprint arXiv:2405.12241. · arXiv
  20. Brinkmann, Jannik, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. 2024. A mechanistic analysis of a transformer trained on a symbolic multi-step reasoning task. arXiv preprint arXiv:2402.11917. · arXiv
  21. Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, Curran Associates, Inc.
  22. Cao, Steven, Victor Sanh, and Alexander Rush. 2021. Low-complexity probing via finding subnetworks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 960-966, Association for Computational Linguistics, Online.
  23. Chen, Angelica, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. 2024a. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. In The Twelfth International Conference on Learning Representations.
  24. Chen, Yida, Aoyu Wu, Trevor DePodesta, Catherine Yeh, Kenneth Li, Nicholas Castillo Marin, Oam Patel, Jan Riecke, Shivam Raval, Olivia Seow, et al. 2024b. Designing a dashboard for transparency and control of conversational AI. arXiv preprint arXiv:2406.07882. · arXiv
  25. Chughtai, Bilal, Lawrence Chan, and Neel Nanda. 2023. A toy model of universality: Reverse engineering how networks learn group operations. In Proceedings of the 40th International Conference on Machine Learning, JMLR.
  26. Cohen, Roi, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2024. Evaluating the Ripple Effects of Knowledge Editing in Language Models. Transactions of the Association for Computational Linguistics, 12:283-298.
  27. Conmy, Arthur, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2023. Towards automated circuit discovery for mechanistic interpretability. In Advances in Neural Information Processing Systems, volume 36, pages 16318-16352,
  28. Conneau, Alexis, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Association for Computational Linguistics, Melbourne, Australia.
  29. Craven, Mark and Jude Shavlik. 1995. Extracting tree-structured representations of trained networks. Advances in Neural Information Processing Systems, 8.
  30. Craven, Mark W. and Jude W. Shavlik. 1994. Using sampling and queries to extract rules from trained neural networks. In William W. Cohen and Haym Hirsh, editors, Machine Learning Proceedings 1994. Morgan Kaufmann, San Francisco (CA), pages 37-45.
  31. Csordás, Róbert, Christopher Potts, Christopher D Manning, and Atticus Geiger. 2024. Recurrent neural networks learn to store and generate sequences using non-linear representations. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 248-262, Association for Computational Linguistics, Miami, Florida, US.
  32. Cunningham, Hoagy, Logan Riggs Smith, Aidan Ewart, Robert Huben, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations.
  33. Dabkowski, Piotr and Yarin Gal. 2017. Real time image saliency for black box classifiers. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Neural Information Processing Systems, page 6970-6979, Curran Associates Inc., Red Hook, NY, USA.
  34. Dalvi, Fahim, Hassan Sajjad, and Nadir Durrani. 2023. NeuroX library for neuron analysis of deep NLP models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 226-234, Association for Computational Linguistics, Toronto, Canada.
  35. Dalvi, Fahim, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. 2020. Analyzing redundancy in pretrained transformer models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4908-4926, Association for Computational Linguistics, Online.
  36. Danilevsky, Marina, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. 2020. A survey of the state of explainable AI for natural language processing. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 447-459, Association for Computational Linguistics, Suzhou, China.
  37. Davies, Xander, Max Nadeau, Nikhil Prakash, Tamar Rott Shaham, and David Bau. 2023. Discovering variable binding circuitry with desiderata. arXiv preprint arXiv:2307.03637. · arXiv
  38. De Cao, Nicola, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. 2020. How do decisions emerge across layers in neural models? Interpretation with differentiable masking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3243-3255, Association for Computational Linguistics, Online.
  39. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Association for Computational Linguistics, Minneapolis, Minnesota.
  40. Dunefsky, Jacob, Philippe Chlenski, and Neel Nanda. 2024. Transcoders find interpretable LLM feature circuits. arXiv preprint 2406.11944. · arXiv
  41. Elazar, Yanai, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175.
  42. Elman, Jeffrey L. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Mach. Learn., 7(2-3):195-225.
  43. Engels, Joshua, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. 2025. Not all language model features are one-dimensionally linear. In The Thirteenth International Conference on Learning Representations.
  44. Erhan, Dumitru, Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2009. Visualizing higher-layer features of a deep network. Technical Report 1341, University of Montreal. Also presented at the ICML 2009 Workshop on Learning Feature Hierarchies, Montréal, Canada.
  45. Fang, Junfeng, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. 2025. Alphaedit: Null-space constrained model editing for language models. In The Thirteenth International Conference on Learning Representations.
  46. Feder, Amir, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. 2022. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond. Transactions of the Association for Computational Linguistics, 10:1138-1158.
  47. Feucht, Sheridan, Eric Todd, Byron Wallace, and David Bau. 2025. The dual-route model of induction. In Second Conference on Language Modeling.
  48. Finlayson, Matthew, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021. Causal analysis of syntactic agreement mechanisms in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1828-1843, Association for Computational Linguistics, Online.
  49. Fiotto-Kaufman, Jaden Fried, Alexander Russell Loftus, Eric Todd, Jannik Brinkmann, Koyena Pal, Dmitrii Troitskii, Michael Ripa, Adam Belfki, Can Rager, Caden Juang, Aaron Mueller, Samuel Marks, Arnab Sen Sharma, Francesca Lucchetti, Nikhil Prakash, Carla E. Brodley, Arjun Guha, Jonathan Bell, Byron C Wallace, and David Bau. 2025. NNsight and NDIF: Democratizing access to open-weight foundation model internals. In The Thirteenth International Conference on Learning Representations.
  50. Gandikota, Rohit, Joanna Materzy ńska, Jaden Fiotto-Kaufman, and David Bau. 2023. Erasing concepts from diffusion models. In Proceedings of the 2023 IEEE International Conference on Computer Vision.
  51. Gandikota, Rohit, Hadas Orgad, Yonatan Belinkov, Joanna Materzy ńska, and David Bau. 2024. Unified concept editing in diffusion models. IEEE/CVF Winter Conference on Applications of Computer Vision.
  52. Geiger, Atticus, Hanson Lu, Thomas F Icard, and Christopher Potts. 2021. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems.
  53. Geiger, Atticus, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. 2024. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pages 160-187, PMLR.
  54. Geva, Mor, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. In The 2023 Conference on Empirical Methods in Natural Language Processing.
  55. Ghorbani, Amirata and James Zou. 2020. Neuron shapley: discovering the responsible neurons. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Curran Associates Inc., Red Hook, NY, USA.
  56. Gilpin, Leilani H, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pages 80-89, IEEE.
  57. Giulianelli, Mario, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 240-248, Association for Computational Linguistics, Brussels, Belgium.
  58. Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, Curran Associates, Inc.
  59. Goyal, Yash, Amir Feder, Uri Shalit, and Been Kim. 2020. Explaining classifiers with causal concept effect (CaCE). arXiv preprint arXiv:1907.07165. · arXiv
  60. Gu, Albert and Tri Dao. 2024. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling.
  61. Gu, Albert, Karan Goel, and Christopher Re. 2022. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations.
  62. Guidotti, Riccardo, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM Comput. Surv., 51(5).
  63. Gupta, Abhijeet, Gemma Boleda, Marco Baroni, and Sebastian Padó. 2015. Distributional vectors encode referential attributes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 12-21, Association for Computational Linguistics, Lisbon, Portugal.
  64. Hanna, Michael, Ollie Liu, and Alexandre Variengien. 2023a. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In Thirty-seventh Conference on Neural Information Processing Systems.
  65. Hanna, Michael, Ollie Liu, and Alexandre Variengien. 2023b. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In Advances in Neural Information Processing Systems, volume 36, pages 76033-76060,
  66. Hanna, Michael, Sandro Pezzelle, and Yonatan Belinkov. 2024. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In ICML 2024 Workshop on Mechanistic Interpretability.
  67. Hase, Peter, Mohit Bansal, Been Kim, and Asma Ghandeharioun. 2023. Does localization inform editing? Surprising differences in causality-based localization vs. knowledge editing in language models. In Thirty-seventh Conference on Neural Information Processing Systems.
  68. Hayashi, Yoichi. 1990. A neural expert system with automated extraction of fuzzy if-then rules and its application to medical diagnosis. In Advances in Neural Information Processing Systems, volume 3, Morgan-Kaufmann.
  69. He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778.
  70. Hewitt, John, Kawin Ethayarajh, Percy Liang, and Christopher Manning. 2021. Conditional probing: Measuring usable information beyond a baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1626-1639, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic.
  71. Hewitt, John and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733-2743, Association for Computational Linguistics, Hong Kong, China.
  72. Hewitt, John and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138, Association for Computational Linguistics, Minneapolis, Minnesota.
  73. Hewitt, John, John Thickstun, Christopher Manning, and Percy Liang. 2023. Backpack language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9103-9125, Association for Computational Linguistics, Toronto, Canada.
  74. Hinton, G. E., J. L. McClelland, and D. E. Rumelhart. 1986. Distributed representations. In Parallel Distributed Processing, Volume 1: Explorations in the Microstructure of Cognition: Foundations. The MIT Press.
  75. Hochreiter, Sepp and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735-1780.
  76. Hu, Xiyang, Cynthia Rudin, and Margo Seltzer. 2019. Optimal sparse decision trees. In Advances in Neural Information Processing Systems, volume 32, pages 7265-7273, Curran Associates, Inc.
  77. Huang, Jing, Atticus Geiger, Karel D'Oosterlinck, Zhengxuan Wu, and Christopher Potts. 2023. Rigorously assessing natural language explanations of neurons. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 317-331, Association for Computational Linguistics, Singapore.
  78. Huang, Jing, Junyi Tao, Thomas Icard, Diyi Yang, and Christopher Potts. 2025. Internal causal mechanisms robustly predict language model out-of-distribution behaviors. In Forty-second International Conference on Machine Learning.
  79. Huang, Jing, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. 2024. RAVEL: Evaluating interpretability methods on disentangling language model representations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8669-8687, Association for Computational Linguistics, Bangkok, Thailand.
  80. Hupkes, Dieuwke, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926.
  81. Hupkes, Dieuwke, Willem Zuidema, et al. 2017. Diagnostic classification and symbolic guidance to understand and improve recurrent neural networks. Interpreting, explaining and visualizing deep learning. Workshop at Neural Information Processing Systems 2017.
  82. Iskander, Shadi, Kira Radinsky, and Yonatan Belinkov. 2023. Shielded representations: Protecting sensitive attributes through iterative gradient-based projection. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5961-5977, Association for Computational Linguistics, Toronto, Canada.
  83. Karnin, E. D. 1990. A simple procedure for pruning back-propagation trained neural networks. IEEE Transactions on Neural Networks, 1(2):239-242.
  84. Karpathy, Andrej, Justin Johnson, and Li Fei-Fei. 2016. Visualizing and understanding recurrent networks. In The Fourth International Conference on Learning Representations.
  85. Karvonen, Adam, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Riggs Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. 2024. Measuring progress in dictionary learning for language model interpretability with board game models. In ICML 2024 Workshop on Mechanistic Interpretability.
  86. Koh, Pang Wei and Percy Liang. 2017. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1885-1894, PMLR.
  87. Koh, Pang Wei, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. 2020. Concept bottleneck models. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5338-5348, PMLR.
  88. Köhn, Arne. 2015. What's in an embedding? Analyzing word embeddings through multilingual evaluation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2067-2073, Association for Computational Linguistics, Lisbon, Portugal.
  89. Kramár, János, Tom Lieberum, Rohin Shah, and Neel Nanda. 2024. AtP*: An efficient and scalable method for localizing llm behaviour to components. arXiv preprint arXiv:2403.00745. · arXiv
  90. Krishnan, R., G. Sivakumar, and P. Bhattacharya. 1999. Extracting decision trees from trained neural networks. Pattern Recognition, 32(12):1999-2009.
  91. Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc.
  92. Lad, Vedang, Wes Gurnee, and Max Tegmark. 2024. The remarkable robustness of LLMs: Stages of inference? In ICML 2024 Workshop on Mechanistic Interpretability.
  93. Lakretz, Yair, Théo Desbordes, Jean-Rémi King, Benoît Crabbé, Maxime Oquab, and Stanislas Dehaene. 2021. Can RNNs learn recursive nested subject-verb agreements? arXiv preprint arXiv:2101.02258. · arXiv
  94. Lakretz, Yair, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 11-20, Association for Computational Linguistics, Minneapolis, Minnesota.
  95. Lasri, Karim, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. 2022. Probing for the usage of grammatical number. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8818-8831, Association for Computational Linguistics, Dublin, Ireland.
  96. Lepori, Michael A., Thomas Serre, and Ellie Pavlick. 2024. Uncovering intermediate variables in transformers using circuit probing. In First Conference on Language Modeling.
  97. Lewis, David. 1986. Causation. In Tim Crane and Katalin Farkas, editors, Philosophical Papers II. Oxford University Press, pages 159-213.
  98. Lewis, David. 2000. Causation as influence. The Journal of Philosophy, 97(4):182-197.
  99. Li, Kenneth, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations.
  100. Li, Maximilian, Xander Davies, and Max Nadeau. 2024. Circuit breaking: Removing model behaviors with targeted ablation. arXiv preprint arXiv:2309.05973. · arXiv
  101. Li, Victoria R., Jenny Kaufmann, Martin Wattenberg, David Alvarez-Melis, and Naomi Saphra. 2025. Can interpretation predict behavior on unseen data? arXiv preprint 2507.06445. · arXiv
  102. Lipton, Zachary C. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31-57.
  103. Liu, Nelson F., Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073-1094, Association for Computational Linguistics, Minneapolis, Minnesota.
  104. Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. · arXiv
  105. Lundberg, Scott M. and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 4768-4777, Curran Associates Inc., Red Hook, NY, USA.
  106. Lyu, Qing, Marianna Apidianaki, and Chris Callison-Burch. 2024. Towards faithful model explanation in NLP: A survey. Computational Linguistics, 50(2):657-723.
  107. Ma, Weicheng, Kai Zhang, Renze Lou, Lili Wang, and Soroush Vosoughi. 2021. Contributions of transformer attention heads in multi-and cross-lingual tasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1956-1966, Association for Computational Linguistics, Online.
  108. Madsen, Andreas, Siva Reddy, and Sarath Chandar. 2022. Post-hoc interpretability for neural NLP: A survey. ACM Comput. Surv., 55(8).
  109. Makelov, Aleksandar, Georg Lange, and Neel Nanda. 2024. Towards principled evaluations of sparse autoencoders for interpretability and control. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models.
  110. Marks, Samuel, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2025. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In The Thirteenth International Conference on Learning Representations.
  111. Marks, Samuel and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824. · arXiv
  112. McClelland, James L. and David E. Rumelhart. 1985. Distributed memory and the representation of general and specific information. Journal of experimental psychology. General, 114 2:159-97.
  113. McDougall, Callum Stuart, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. 2024. Copy suppression: Comprehensively understanding a motif in language model attention heads. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 337-363, Association for Computational Linguistics, Miami, Florida, US.
  114. McGrath, Thomas, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. 2023. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2307.15771. · arXiv
  115. Méloux, Maxime, Silviu Maniu, François Portet, and Maxime Peyrard. 2025. Everything, everywhere, all at once: Is mechanistic interpretability identifiable? In The Thirteenth International Conference on Learning Representations.
  116. Meng, Kevin, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems.
  117. Meng, Kevin, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations.
  118. Merullo, Jack, Carsten Eickhoff, and Ellie Pavlick. 2024a. Circuit component reuse across tasks in transformer language models. In The Twelfth International Conference on Learning Representations.
  119. Merullo, Jack, Carsten Eickhoff, and Ellie Pavlick. 2024b. Talking heads: Understanding inter-layer communication in transformer language models. arXiv preprint arXiv:2406.09519. · arXiv
  120. Michaud, Eric J, Ziming Liu, Uzay Girit, and Max Tegmark. 2023. The quantization model of neural scaling. In Thirty-seventh Conference on Neural Information Processing Systems.
  121. Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. · arXiv
  122. Mikolov, Tomas, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Interspeech, pages 1045-1048.
  123. Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26, Curran Associates, Inc.
  124. Miller, Joseph, Bilal Chughtai, and William Saunders. 2024. Transformer circuit evaluation metrics are not robust. In First Conference on Language Modeling.
  125. Mohebbi, Hosein, Jaap Jumelet, Michael Hanna, Afra Alishahi, and Willem Zuidema. 2024. Transformer-specific interpretability. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts, pages 21-26, Association for Computational Linguistics, St. Julian's, Malta.
  126. Moraffah, Raha, Mansooreh Karami, Ruocheng Guo, Adrienne Raglin, and Huan Liu. 2020. Causal interpretability for machine learning-problems, methods and evaluation. ACM SIGKDD Explorations Newsletter, 22(1):18-33.
  127. Morcos, Ari S., David G.T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. 2018. On the importance of single directions for generalization. In International Conference on Learning Representations.
  128. Mozer, Michael C. and Paul Smolensky. 1988. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In Advances in Neural Information Processing Systems, volume 1, Morgan-Kaufmann.
  129. Mueller, Aaron. 2024. Missed causes and ambiguous effects: Counterfactuals pose challenges for interpreting neural networks. In ICML 2024 Workshop on Mechanistic Interpretability.
  130. Mueller, Aaron, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yaniv Nikankin, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessandro Stolfo, Martin Tutek, Amir Zur, David Bau, and Yonatan Belinkov. 2025. Mib: A mechanistic interpretability benchmark. In Proceedings of the 42nd International Conference on Machine Learning, PMLR.
  131. Mueller, Aaron, Yu Xia, and Tal Linzen. 2022. Causal analysis of syntactic agreement neurons in multilingual language models. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 95-109, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid).
  132. Nanda, Neel, Andrew Lee, and Martin Wattenberg. 2023. Emergent linear representations in world models of self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 16-30, Association for Computational Linguistics, Singapore.
  133. Neo, Clement, Shay B Cohen, and Fazl Barez. 2024. Interpreting context look-ups in transformers: Investigating attention-MLP interactions. arXiv preprint arXiv:2402.15055. · arXiv
  134. Odense, Simon and Artur d'Avila Garcez. 2020. Layerwise knowledge extraction from deep convolutional networks. arXiv preprint arXiv:2003.09000. · arXiv
  135. Oikarinen, Tuomas, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. 2023. Label-free concept bottleneck models. In The Eleventh International Conference on Learning Representations.
  136. Panickssery, Nina, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2024. Steering Llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681. · arXiv
  137. Park, Kiho, Yo Joong Choe, and Victor Veitch. 2023. The linear representation hypothesis and the geometry of large language models. In Causal Representation Learning Workshop at NeurIPS 2023.
  138. Paulo, Gonçalo, Alex Mallen, Caden Juang, and Nora Belrose. 2024. Automatically interpreting millions of features in large language models. arXiv preprint arXiv:2410.13928. · arXiv
  139. Pearl, Judea. 2000. Causality: Models, Reasoning, and Inference. Cambridge University Press.
  140. Pearl, Judea. 2001. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, pages 411-420, Morgan Kaufmann.
  141. Pearl, Judea. 2009. Causality, 2 edition. Cambridge University Press.
  142. Pennington, Jeffrey, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Association for Computational Linguistics, Doha, Qatar.
  143. Prakash, Nikhil, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. 2024. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In The Twelfth International Conference on Learning Representations.
  144. Qiu, Linlu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. 2024. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. In The Twelfth International Conference on Learning Representations.
  145. Rai, Daking, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. 2024. A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646. · arXiv
  146. Ravfogel, Shauli, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Association for Computational Linguistics, Online.
  147. Ravfogel, Shauli, Grusha Prasad, Tal Linzen, and Yoav Goldberg. 2021. Counterfactual interventions reveal the causal effect of relative clause representations on agreement prediction. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 194-209, Association for Computational Linguistics, Online.
  148. Ravfogel, Shauli, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. 2022. Adversarial concept erasure in kernel space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6034-6055, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates.
  149. Ravichander, Abhilasha, Yonatan Belinkov, and Eduard Hovy. 2021. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3363-3377, Association for Computational Linguistics, Online.
  150. Rawal, Atul, Adrienne Raglin, Danda B. Rawat, Brian M. Sadler, and James McCoy. 2024. Causality for trustworthy artificial intelligence: Status, challenges and perspectives. ACM Comput. Surv.
  151. Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. 2016a. Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386. · arXiv
  152. Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. 2016b. "Why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, page 1135-1144, Association for Computing Machinery, New York, NY, USA.
  153. Robins, James M. and Sander Greenland. 1992. Identifiability and exchangeability for direct and indirect effects. Epidemiology, 3(2):143-155.
  154. Robnik-Šikonja, Marko and Igor Kononenko. 2008. Explaining classifications for individual instances. IEEE Transactions on Knowledge and Data Engineering, 20(5):589-600.
  155. Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. 1986. Learning representations by back-propagating errors. Nature, 323(6088):533-536.
  156. Sajjad, Hassan, Nadir Durrani, and Fahim Dalvi. 2022. Neuron-level interpretation of deep NLP models: A survey. Transactions of the Association for Computational Linguistics, 10:1285-1303.
  157. Salmon, Wesley C. 1984. Scientific Explanation and the Causal Structure of the World. Princeton University Press.
  158. Saphra, Naomi, Eve Fleisig, Kyunghyun Cho, and Adam Lopez. 2024. First tragedy, then parse: History repeats itself in the new era of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2310-2326, Association for Computational Linguistics, Mexico City, Mexico.
  159. Saphra, Naomi and Sarah Wiegreffe. 2024. Mechanistic? In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 480-498, Association for Computational Linguistics, Miami, Florida, US.
  160. Shaham, Tamar Rott, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob Andreas, and Antonio Torralba. 2024. A multimodal automated interpretability agent. In Forty-first International Conference on Machine Learning.
  161. Sharma, Arnab Sen, David Atkinson, and David Bau. 2024. Locating and editing factual associations in Mamba. arXiv preprint arXiv:2404.03646. · arXiv
  162. Shi, Xing, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526-1534, Association for Computational Linguistics, Austin, Texas.
  163. Simonyan, Karen, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034. · arXiv
  164. Singh, Chandan, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. 2024a. Rethinking interpretability in the era of large language models. arXiv preprint arXiv:2402.01761. · arXiv
  165. Singh, Shashwat, Shauli Ravfogel, Jonathan Herzig, Roee Aharoni, Ryan Cotterell, and Ponnurangam Kumaraguru. 2024b. Representation surgery: theory and practice of affine steering. In Proceedings of the 41st International Conference on Machine Learning, ICML'24, JMLR.org.
  166. Srivastava, Divyansh, Tuomas Oikarinen, and Tsui-Wei Weng. 2023. Corrupting neuron explanations of deep visual features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1877-1886.
  167. Strobelt, Hendrik, Sebastian Gehrmann, Hanspeter Pfister, and Alexander M Rush. 2017. LSTMVis: A tool for visual analysis of hidden state dynamics in recurrent neural networks. IEEE Transactions on Visualization and Computer Graphics, 24(1):667-676.
  168. Subramani, Nishant, Nivedita Suresh, and Matthew Peters. 2022. Extracting latent steering vectors from pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 566-581, Association for Computational Linguistics, Dublin, Ireland.
  169. Sun, Jiuding, Jing Huang, Sidharth Baskaran, Karel D'Oosterlinck, Christopher Potts, Michael Sklar, and Atticus Geiger. 2025. HyperDAS: Towards automating mechanistic interpretability with hypernetworks. In The Thirteenth International Conference on Learning Representations.
  170. Sundararajan, Mukund, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning -Volume 70, ICML'17, page 3319-3328, JMLR.org.
  171. Syed, Aaquib, Can Rager, and Arthur Conmy. 2023. Attribution patching outperforms automated circuit discovery. In NeurIPS Workshop on Attributing Model Behavior at Scale.
  172. Tamkin, Alex, Mohammad Taufeeque, and Noah D. Goodman. 2023. Codebook features: Sparse and discrete interpretability for neural networks. arXiv preprint arXiv:2310.17230. · arXiv
  173. Theodosis, Emmanouil and Demba Ba. 2023. Learning silhouettes with group sparse autoencoders. In ICASSP 2023 -2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
  174. Tigges, Curt, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. 2023. Linear representations of sentiment in large language models. arXiv preprint arXiv:2310.15154. · arXiv
  175. Todd, Eric, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. 2024. Function vectors in large language models. In The Twelfth International Conference on Learning Representations.
  176. Tucker, Mycal, Peng Qian, and Roger Levy. 2021. What if this modified that? Syntactic interventions with counterfactual embeddings. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 862-875, Association for Computational Linguistics, Online.
  177. Turner, Alexander Matt, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. 2023. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248. · arXiv
  178. Tzeng, F.-Y. and K.-L. Ma. 2005. Opening the black box -Data driven visualization of neural networks. In VIS 05. IEEE Visualization, 2005., pages 383-390.
  179. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc.
  180. Vig, Jesse, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pages 12388-12401,
  181. Wang, Kevin Ro, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations.
  182. White, Adam and Artur d'Avila Garcez. 2020. Measurable counterfactual local explanations for any classifier. In European Conference on Artificial Intelligence. IOS Press, pages 2529-2535.
  183. Wu, Wenhao, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. 2024a. Retrieval head mechanistically explains long-context factuality. arXiv preprint arXiv:2404.15574. · arXiv
  184. Wu, Xing, Shaoqi Peng, Jingwen Li, Jian Zhang, Qun Sun, Weimin Li, Quan Qian, Yue Liu, and Yike Guo. 2024b. Causal inference in the medical domain: A survey. Applied Intelligence, pages 1-24.
  185. Wu, Zhengxuan, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. 2025. Axbench: Steering llms? even simple Computational Linguistics Just Accepted MS. https://doi.org/10.1162/coli.a.561 © 2025 Association for Computational Linguistics Published under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license Computational Linguistics Volume vv, Number nn · DOI
  186. Wu, Zhengxuan, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2024c. ReFT: Representation finetuning for language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  187. Wu, Zhengxuan, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah Goodman, Christopher Manning, and Christopher Potts. 2024d. pyvene: A library for understanding and improving PyTorch models via interventions. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations), pages 158-165, Association for Computational Linguistics, Mexico City, Mexico.
  188. Wu, Zhengxuan, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. 2023. Interpretability at scale: Identifying causal mechanisms in alpaca. In Thirty-seventh Conference on Neural Information Processing Systems.
  189. Zeiler, Matthew D. and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Computer Vision -ECCV 2014, pages 818-833, Springer International Publishing, Cham.
  190. Zhong, Zexuan, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. 2023. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15686-15702.

Cited by

Sorted by citations

See all 1

Metadata

Authors
Mueller, A., Brinkmann, J., Li, M., Marks, S., Pal, K., Prakash, N., Rager, C., Sankaranarayanan, A., Sharma, A. S., Sun, J., Todd, E., Bau, D., & Belinkov, Y.
Venue
arXiv
Publisher
arXiv
Published
2024
DOI
10.48550/arxiv.2408.01416
arXiv
2408.01416
References
2
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review