CoChat
Sign Up
PreprintCited by 291Explainable Artificial Intelligence (XAI)Neural Networks and ApplicationsAdversarial Robustness in Machine Learning

The DOI matches its DataCite record; 232 of the 232 references it cites were found in a registry or an index; 67 of the 249 sources its claims cite were checked for support.

Open Problems in Mechanistic Interpretability

Open access
Sharkey, Lee, Chughtai, Bilal, Batson, Joshua, Lindsey, Jack, Wu, Jeff, Bushnaq, Lucius, Goldowsky-Dill, Nicholas, Heimersheim, Stefan, Ortega, Alejandro, Bloom, Joseph, Biderman, Stella, Garriga-Alonso, Adria, Conmy, Arthur, Nanda, Neel, Rumbelow, Jessica, Wattenberg, Martin, Schoots, Nandi, Miller, Joseph, Michaud, Eric J., Casper, Stephen, Tegmark, Max, Saunders, William, Bau, David, Todd, Eric, Geiger, Atticus, Geva, Mor, Hoogland, Jesse, Murfet, Daniel, McGrath, Tom·2025·arXiv
DOI 10.48550/arxiv.2501.16496arXiv 2501.16496
Open this paper in CoChat — free accountOpen PDF

Abstract

Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks' capabilities in order to accomplish concrete scientific and engineering goals. Progress in this field thus promises to provide greater assurance over AI system behavior and shed light on exciting scientific questions about the nature of intelligence. Despite recent progress toward these goals, there are many open problems in the field that require solutions before many scientific and practical benefits can be realized: Our methods require both conceptual and practical improvements to reveal deeper insights; we must figure out how best to apply our methods in pursuit of specific goals; and the field must grapple with socio-technical challenges that influence and are influenced by our work. This forward-facing review discusses the current frontier of mechanistic interpretability and the open problems that the field may benefit from prioritizing.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This forward-facing review surveys open problems in mechanistic interpretability across methods/foundations, applications, and socio-technical challenges. It argues that progress requires better decomposition, description, validation, application-specific methods, and governance engagement before the field’s scientific and engineering goals can be realized.

Key findings

  • Current decomposition methods, especially sparse dictionary learning (SDL), face major practical and conceptual limitations: high reconstruction errors, scaling costs, reliance on the linear representation hypothesis, sparsity as an imperfect proxy for interpretability, unexplained feature geometry, architecture-specific difficulties, and inability to describe mechanisms directly.
  • Applications could include monitoring/auditing unsafe cognition, controlling model behavior via steering/unlearning/editing/finetuning, predicting novel behavior and capabilities, improving inference/training, microscope AI, and human-computer interaction with model internals; however, interpretability has produced few tools used to make state-of-the-art systems safer in the real world.
  • The paper highlights cited quantitative examples: inserting a sparse dictionary with 16 million latents into GPT-4 yielded language modeling loss equivalent to a model with only 10% of GPT-4’s pretraining compute; SAE reconstructions decreased GPT-2 small performance by 10% on task-specific data and 40% on the full distribution.
  • Socio-technical open problems include translating technical progress into AI policy/governance levers, clarifying the field’s goals and success criteria, comparing interpretability against non-interpretability baselines, and mitigating misuse or false assurance risks.

Methods

  • Design: a forward-facing review and synthesis of perspectives from its various authors, coordinated by ApolloResearch on behalf of Schmidt Sciences; it is not an empirical study.
  • Data or sample: no new dataset or experiments; it draws on existing mechanistic interpretability literature and cited works.
  • Main analysis: conceptual organization of open problems into methods/foundations (reverse engineering, concept-based interpretability, circuit discovery, automation), applications, and socio-technical challenges.

Limitations

  • Stated: the perspectives presented do not necessarily reflect the views of any individual author or the institutions with which they are affiliated; the review collects and synthesizes authors’ views by ApolloResearch on behalf of Schmidt Sciences.
  • Evident: as a review/position synthesis, it provides no new empirical validation, systematic search protocol, or quantitative meta-analysis; its claims about open problems are author perspectives rather than tested findings.
  • Evident: many quantitative results cited are from other papers and are not independently reproduced here.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2501.16496

Cites

232 of 232 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Two approaches to neural network interpretability. (Left) Reverse Engineering is characterized by decomposing networks into functional components and describing how those components interact to produce the network's behavior. It thus aims to 'identify the roles of network components' (Section 2.1). (Right) Concept-based interpretability on the other hand attempts to discover human concepts within neural network internals. It thus aims to 'identify the network components for given roles' (Section 2.2).
Figure 1 Two approaches to neural network interpretability. (Left) Reverse Engineering is characterized by decomposing networks into functional components and describing how those components interact to produce the network's behavior. It thus aims to 'identify the roles of network components' (Section 2.1). (Right) Concept-based interpretability on the other hand attempts to discover human concepts within neural network internals. It thus aims to 'identify the network components for given roles' (Section 2.2).CC-BY · From the PDF
Figure 2 The steps of reverse engineering neural networks. (1) Decomposing a network into simpler components. This decomposition might not necessarily use architecturally-defined bases, such as individual neurons or layers (Section 2.1.2). (2) Hypothesizing about the functional roles of some or all components (Section 2.1.3). (3) Validating whether our hypotheses are correct, creating a cycle in which we iteratively refine our decompositions and hypotheses to improve our understanding of the network (Section 2.1.4).
Figure 2 The steps of reverse engineering neural networks. (1) Decomposing a network into simpler components. This decomposition might not necessarily use architecturally-defined bases, such as individual neurons or layers (Section 2.1.2). (2) Hypothesizing about the functional roles of some or all components (Section 2.1.3). (3) Validating whether our hypotheses are correct, creating a cycle in which we iteratively refine our decompositions and hypotheses to improve our understanding of the network (Section 2.1.4).CC-BY · From the PDF
Figure 3 Three ideas underlying the sparse dictionary learning (SDL) paradigm in mechanistic interpretability. (Left) The linear representation hypothesis states that the map from 'concepts' to neural activations is linear. (Middle) Superposition is the hypothesis that models represent many more concepts than they have dimensions by representing them both sparsely and linearly in activation spaces. (Right) SDL attempts to recover an overcomplete basis of concepts represented in superposition in activation space.
Figure 3 Three ideas underlying the sparse dictionary learning (SDL) paradigm in mechanistic interpretability. (Left) The linear representation hypothesis states that the map from 'concepts' to neural activations is linear. (Middle) Superposition is the hypothesis that models represent many more concepts than they have dimensions by representing them both sparsely and linearly in activation spaces. (Right) SDL attempts to recover an overcomplete basis of concepts represented in superposition in activation space.CC-BY · From the PDF
Figure 4 Sparse dictionary learning has a number of practical and conceptual limitations that cause issues when using it to reverse engineer neural networks (Paragraph 2.1.2c).
Figure 4 Sparse dictionary learning has a number of practical and conceptual limitations that cause issues when using it to reverse engineer neural networks (Paragraph 2.1.2c).CC-BY · From the PDF
Figure 5 To study the functional role of the blue component numerous approaches are possible. We could study its causes: the purple input components or the red intermediate components via e.g. feature synthesis (Olah et al., 2017a; 2020b), maximum activating examples (Olah et al., 2017a; Bricken, 2023), or attributions (Sundararajan et al., 2017). Or we could study its effects: the orange output components or the green intermediate components via e.g. the logit lens (Nostalgebraist, 2020), activation steering (Turner et al., 2024), or attributions (Marks et al., 2024).
Figure 5 To study the functional role of the blue component numerous approaches are possible. We could study its causes: the purple input components or the red intermediate components via e.g. feature synthesis (Olah et al., 2017a; 2020b), maximum activating examples (Olah et al., 2017a; Bricken, 2023), or attributions (Sundararajan et al., 2017). Or we could study its effects: the orange output components or the green intermediate components via e.g. the logit lens (Nostalgebraist, 2020), activation steering (Turner et al., 2024), or attributions (Marks et al., 2024).CC-BY · From the PDF
Figure 6 A summary of problem areas for applications of mechanistic interpretability.
Figure 6 A summary of problem areas for applications of mechanistic interpretability.CC-BY · From the PDF

Verification

Claims the paper makes

80 Sources not found · 76 Own interpretation · 41 Partly supported · 10 Supported · 9 No basis given · 2 Couldn’t confirm · 1 Supported so far · 1 No full text of the sources · 1 Not checked yet · accuracy not yet measured

Verify all 3 references — free account
  • Probably not a substantial claim. 85
  • It may state two separate claims as one. 39
  • The quote only partly states this claim. 1
110 of 110

  • The paper relies on this without support.

    Many attempts to incentivize interpretable activations directly have not been competitively performant and have allowed 'superposition to sneak through', mitigating benefits.

    “Many attempts to incentivize interpretable activations directly so far have not been competitively performant, and have also allowed 'superposition to sneak through', mitigating benefits.”

  • The paper relies on this without support.

    The current strategy of training a model solely for performance and then interpreting it post hoc may not be optimal if our goal is a model that is both interpretable and performant.

    “The current strategy of training a model solely for performance and then interpreting it post hoc may not be optimal if our goal is a model that is both interpretable and performant.”

  • The paper relies on this without support.

    Currently, interpretability researchers lack consensus on which networks should serve as model organisms.

    “Currently, interpretability researchers lack consensus on which networks should serve as model organisms.”

  • The paper relies on this without support.

    The practice of conducting evaluations using non-cherry-picked tasks remains relatively uncommon.

    “However, the practice of conducting evaluations using non-cherry-picked tasks remains relatively uncommon.”

  • The paper relies on this without support.

    It is often hard to distinguish faithful explanations of neural network components from merely plausible ones.

    “Unfortunately, it is often hard to distinguish faithful explanations of neural network components from merely plausible ones.”

  • The paper relies on this without support.

    Interpretability researchers have historically faced challenges in adequately validating their hypotheses due to the high costs in terms of time and cognitive labor.

    “Interpretability researchers have historically faced challenges in adequately validating their hypotheses due to the high costs in terms of time and cognitive labor.”

  • The authors state this as an assumption.

    Developers may have white-box access to models, while users may not.

    “One reasonable assumption is that developers may have white-box access to models, while users may not.”

  • The authors state this as an assumption.

    A decomposition method that carves networks at the joints of their generalization structure would be maximally interpretable.

    “This was previously used to measure progress toward a decomposition method that carves networks at the joints of their generalization structure, assuming that such a decomposition would be maximally interpretable (Section 2.1.2).”

  • The paper relies on this without support.

    Current mechanistic interpretability research is primarily focused on understanding mechanisms involved in predictions at a single time step.

    “However, current mechanistic interpretability research is primarily focused on understanding mechanisms involved in predictions at a single time step.”

  • The paper relies on this without support.

    Capabilities are behaviors that often span multiple sequential steps.

    “Since capabilities are behaviors that often span multiple sequential steps, it may be necessary to have mechanistic interpretability methods that examine mechanisms spanning multiple time steps.”

  • The paper relies on this without support.

    Modern AI systems are still generally trained, evaluated, monitored, and debugged using techniques that do not rely on understanding their internal workings.

    “Modern AI systems are still generally trained, evaluated, monitored, and debugged using techniques that do not rely on understanding their internal workings.”

  • The paper relies on this without support.

    Interpretability-based methods do not currently outperform black-box baselines.

    “While interpretability-based methods do not currently outperform black-box baselines, if they perform in a similar ballpark, further progress in mechanistic interpretability could soon lead to better methods, especially in problems which we believe might disproportionately benefit from improved mechanistic understanding (Section 3).”

  • The paper relies on this without support.

    The field of AI interpretability is highly influenced by research teams in the private sector.

    “The field of AI interpretability is highly influenced by research teams in the private sector.”

  • The paper relies on this without support.

    Compared to academia, corporations publish selectively.

    “On the other hand, compared to academia, corporations publish selectively, often have financial conflicts of interest, and may provide limited transparency.”

  • The paper relies on this without support.

    Corporations often have financial conflicts of interest.

    “On the other hand, compared to academia, corporations publish selectively, often have financial conflicts of interest, and may provide limited transparency.”

  • The paper relies on this without support.

    Corporations may provide limited transparency.

    “On the other hand, compared to academia, corporations publish selectively, often have financial conflicts of interest, and may provide limited transparency.”

  • The paper relies on this without support.

    Deep learning models are capable of encoding complex, non-linear relationships and extracting meaningful features from massive data sets without human intervention.

    “Deep learning models are capable of encoding complex, non-linear relationships and extracting meaningful features from massive data sets without human intervention.”

  • The paper relies on this without support.

    Current approaches for knowledge discovery from data involve statistical or causal analysis, dimensionality reduction, or using machine learning models that are inherently interpretable.

    “Current approaches for knowledge discovery from data involve statistical or causal analysis, dimensionality reduction, or using machine learning models that are inherently interpretable.”

  • The paper relies on this without support.

    These techniques are influenced by human priors, typically assume linear relationships between variables, and cannot handle massive multimodal data.

    “These techniques can be valuable, but are influenced by human priors, typically assume linear relationships between variables, and cannot handle massive multimodal data.”

  • The paper relies on this without support.

    Currently, the majority of scientists are unable to access microscope AI due to the need for specialized expertise in machine learning, interpretability, and domain knowledge to recognize significant new patterns, a combination of skills that is rare in many fields.

    “Currently, the majority of scientists are unable to access microscope AI due to the need for specialized expertise in machine learning, interpretability, and domain knowledge to recognize significant new patterns, a combination of skills that is rare in many fields.”

  • The paper relies on this without support.

    Industry resources and research contributions have significantly advanced interpretability research.

    “On one hand, industry resources and research contributions have significantly advanced interpretability research.”

  • The paper relies on this without support.

    The black-box nature of AI models remains unresolved.

    “While the black-box nature of AI models remains unresolved, the untapped potential of mechanistic interpretability is what makes it such an exciting research area, and highlights the importance of solving its many open research problems.”

  • The paper relies on this without support.

    The EU AI Act mandates that developers of General Purpose AI models with systemic risk have obligations to report incidents involving their system to the AI office.

    “The EU AI Act mandates that developers of General Purpose AI models with systemic risk have obligations to report incidents involving their system to the AI office.”
23 of 23

Citations

Cites

Part of this paper’s reference list could not be split into single entries.

Show all 197 references

Research works printed but not linked: 157

  1. Abubakar Abid, Mert Yuksekgonul, and James Zou. Meaningfully debugging model mistakes using concep- tual counterfactual explanations. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learn- ing, volume 162 of Proceedings of Machine Learning Research, pp. 66-88. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/abid22a.html.
  2. Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. San- ity checks for saliency maps. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/ 294a8ed24b1ad22ec2e7efea049b8737-Paper.pdf.
  3. Julius Adebayo, Michael Muelly, Ilaria Liccardi, and Been Kim. Debugging tests for model explanations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 700-712.
  4. Curran Associates, Inc., 2020. URL https://proceedings. neurips.cc/paper_files/paper/2020/file/075b051ec3d22dac7b33f788da631fd4-Paper.pdf.
  5. Chirag Agarwal, Sree Harsha Tanneru, and Himabindu Lakkaraju. Faithfulness vs. plausibility: On the (un)reliability of explanations from large language models, 2024. URL https://arxiv.org/abs/2402. 04614.
  6. Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neu- ral nets. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 4699-4711.
  7. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ 251bd0442dfcc53b5a761e050f8022b8-Paper.pdf.
  8. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2017. URL https://openreview.net/forum?id=ryF7rTqgl.
  9. Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric J Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Chenyu Zhang, Ruiqi Zhong, Sean O hEigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Aleksandar Petrov, Christian Schroeder de Witt, Sumeet Ramesh Motwani, Yoshua Bengio, Danqi Chen, Philip Torr, Samuel Albanie, Tegan Ma- haraj, Jakob Nicolaus Foerster, Florian Tramèr, He He, Atoosa Kasirzadeh, Yejin Choi, and David Krueger. Foundational challenges in assuring alignment and safety of large language models. Transactions on Ma- chine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=oVTkOs8Pka. Survey Certification, Expert Certification.
  10. Randall Balestriero and richard baraniuk. A spline theory of deep learning. In Jennifer Dy and An- dreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 374-383. PMLR, 10-15 Jul 2018. URL https: //proceedings.mlr.press/v80/balestriero18b.html.
  11. Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48 (1):207-219, March 2022b. doi: 10.1162/coli_a_00422. URL https://aclanthology.org/2022.cl-1. · DOI
  12. Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 66044-66063.
  13. Curran Associates, Inc., 2023b. URL https://proceedings.neurips.cc/paper_files/ paper/2023/file/d066d21c619d0a78c5b557fa3291a8f4-Paper-Conference.pdf.
  14. Leonard Bereska and Stratis Gavves. Mechanistic interpretability for AI safety -a review. Transac- tions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum? id=ePUVetPKu6. Survey Certification, Expert Certification.
  15. Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMs trained on "a is b" fail to learn "b is a". In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id= GPKTIktA0k.
  16. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https: //openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023.
  17. Judy Borowski, Roland Simon Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain {cnn} activations better than state-of-the-art feature visualization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=QO9-y8also-.
  18. Leo Breiman. Classification and regression trees. Routledge, 1984.
  19. Wieland Brendel and Matthias Bethge. Approximating CNNs with bag-of-local-features models works surprisingly well on imagenet. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=SkfMWhAqYQ.
  20. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=ETKGuby0hcs.
  21. Stephen Casper, Max Nadeau, Dylan Hadfield-Menell, and Gabriel Kreiman. Robust feature-level adver- saries are interpretability tools. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 33093-33106.
  22. Stephen Casper, Max Nadeau, Dylan Hadfield-Menell, and Gabriel Kreiman. Robust feature-level adver- saries are interpretability tools. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 33093-33106.
  23. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomek Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel- Raphael Segerie, Micah Carroll, Andi Peng, Phillip J.K. Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michaud, Jacob Pfau, Dmitrii Krashenin- nikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental limitations of reinforcement learn- ing from human feedback. Transactions on Machine Learning Research, 2023b. ISSN 2835-8856. URL https://openreview.net/forum?id=bx24KpJ4Eb. Survey Certification, Featured Certification.
  24. Haozhe Chen, Carl Vondrick, and Chengzhi Mao. Selfie: self-interpretation of large language model embed- dings. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2025.
  25. Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/ 1ee3dfcd8a0645a25a35977997223d22-Paper.pdf.
  26. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/ 2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf.
  27. Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara En- gelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 6243-6267. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/chughtai23a.html.
  28. Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 16318-16352.
  29. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/ paper/2023/file/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Paper-Conference.pdf.
  30. Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision -ECCV 2022, pp. 88-105, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031- 19836-6.
  31. Adrián Csiszárik, Péter Kőrösi-Szabó, Ákos K. Matszangosz, Gergely Papp, and Dániel Varga. Similarity and matching of neural network representations. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview. net/forum?id=aedFIIRRfXr.
  32. Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. In International Conference on Learning Rep- resentations, 2021. URL https://openreview.net/forum?id=7uVcpu-gMD.
  33. Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural In- formation Processing Systems, volume 32.
  34. Curran Associates, Inc., 2019. URL https://proceedings. neurips.cc/paper_files/paper/2019/file/bb836c01cdc9120a9c984c525e4b1a4a-Paper.pdf.
  35. Jonathan Donnelly and Adam Roegiest. On interpretability and feature representations: An analysis of the sentiment neuron. In Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia Hauff, and Djoerd Hiemstra (eds.), Advances in Information Retrieval, pp. 795-802, Cham, 2019. Springer International Publishing. ISBN 978-3-030-15712-8.
  36. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy.
  37. Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: //openreview.net/forum?id=J6zHcScAo0.
  38. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022. URL http://jmlr.org/papers/v23/21-0998.html.
  39. Jakob N. Foerster, Justin Gilmer, Jascha Sohl-Dickstein, Jan Chorowski, and David Sussillo. Input switched affine networks: an rnn architecture designed for interpretability. In Proceedings of the 34th International Conference on Machine Learning -Volume 70, ICML'17, pp. 1136-1145. JMLR.org, 2017.
  40. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https://openreview. net/forum?id=rJl-b3RcF7.
  41. Timo Freiesleben and Gunnar König. Dear xai community, we need to talk! In Luca Longo (ed.), Explainable Artificial Intelligence, pp. 48-65, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-44064-9.
  42. Dan Friedman, Andrew Kyle Lampinen, Lucas Dixon, Danqi Chen, and Asma Ghandeharioun. Interpretabil- ity illusions in the generalization of simplified models, 2024. URL https://openreview.net/forum?id= v675Iyu0ta.
  43. Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=COZDy0WYGg.
  44. Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting CLIP's image representation via text-based decomposition. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=5Ca9sSzuDp.
  45. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024. URL https://arxiv.org/abs/ 2406.04093.
  46. Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 7324-7338. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/geiger22a.html.
  47. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2024b.
  48. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Francesco Locatello and Vanessa Didelez (eds.), Proceedings of the Third Conference on Causal Learning and Reasoning, volume 236 of Proceedings of Machine Learning Research, pp. 160-187. PMLR, 01-03 Apr 2024c. URL https://proceedings.mlr.press/v236/geiger24a.html.
  49. Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019. URL https://openreview. net/forum?id=Bygh9j09KX.
  50. Robert Geirhos, Roland S. Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don't trust your eyes: on the (un)reliability of feature visualizations. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2025.
  51. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: a unifying framework for inspecting hidden representations of language models. In Proceedings of the 41st Interna- tional Conference on Machine Learning, ICML'24. JMLR.org, 2025.
  52. Amirata Ghorbani and James Y Zou. Neuron shapley: Discovering the responsible neurons. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5922-5932.
  53. Curran Associates, Inc., 2020. URL https://proceedings.neurips. cc/paper_files/paper/2020/file/41c542dfe6e4fc3deb251d64cf6ed2e4-Paper.pdf.
  54. Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov, and David Krueger. Stress-testing capability elic- itation with password-locked models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=zzOOqD6R1b.
  55. Phillip Huang Guo, Aaquib Syed, Abhay Sheshadri, Aidan Ewart, and Gintare Karolina Dziugaite. Robust unlearning via mechanistic localizations. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. URL https://openreview.net/forum?id=06pNzrEjnH.
  56. Rohan Gupta, Iván Arcuschin, Thomas Kwa, and Adrià Garriga-Alonso. Interpbench: Semi-synthetic trans- formers for evaluating mechanistic interpretability techniques. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/ forum?id=R9gR9MPuD5.
  57. Wes Gurnee and Max Tegmark. Language models represent space and time. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=jE8xbmvFin.
  58. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=JYs1R9IMJr.
  59. Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems -Volume 1, NIPS'15, pp. 1135-1143, Cambridge, MA, USA, 2015. MIT Press.
  60. Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In Thirty-seventh Conference on Neural Informa- tion Processing Systems, 2023. URL https://openreview.net/forum?id=p4PckNQR8k.
  61. Kaarel Hänni, Jake Mendel, Dmitry Vaintrob, and Lawrence Chan. Mathematical models of computation in superposition. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. URL https://openreview. net/forum?id=OcVJP8kClR.
  62. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id= EldbUlZtbd.
  63. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10.1109/CVPR.2016. · DOI
  64. Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversar- ial model manipulation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32.
  65. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/ 7fea637fd6d02b8f0adf6f7dc36aed93-Paper.pdf.
  66. Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep features. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=NudBMY-tzDr.
  67. Geoffrey Hinton. Shape representation in parallel systems. Proceedings of teh Seventh International Joint Conference on Artificial Intelligence, 1981. URL https://www.cs.toronto.edu/~hinton/absps/ shape81.pdf.
  68. Xiyang Hu, Cynthia Rudin, and Margo I. Seltzer. Optimal sparse decision trees. In Neural Information Processing Systems, 2019. URL https://api.semanticscholar.org/CorpusID:139104649.
  69. Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=F76bwRSLeK.
  70. Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/ 5a4be1fa34e62bb8a6ec6b91d2462f5a-Paper.pdf.
  71. Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Tim Rocktäschel, Edward Grefenstette, and David Krueger. Mechanistically analyzing the effects of fine-tuning on proce- durally defined tasks. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=A0HKeKl4Nl.
  72. Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3543-3556, Minneapolis, Minne@inproceedingskatz-etal-2024-backward, title = "Backward Lens: Pro- jecting Language Model Gradients into the Vocabulary Space", author = "Katz, Shahar and Belinkov, Yonatan and Geva, Mor and Wolf, Lior", editor = "Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung", booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing", month = nov, year = "2024", address = "Miami, Florida, USA", publisher = "As- sociation for Computational Linguistics", url = "https://aclanthology.org/2024.emnlp-main.142/", doi = "10.18653/v1/2024.emnlp-main.142", pages = "2390-2422", abstract = "Understanding how Transformer- based Language Models (LMs) learn and recall information is a key goal of the deep learning community. Recent interpretability methods project weights and hidden states obtained from the forward pass to the models' vocabularies, helping to uncover how information flows within LMs. In this work, we ex- tend this methodology to LMs' backward pass and gradients. We first prove that a gradient matrix can be cast as a low-rank linear combination of its forward and backward passes' inputs. We then develop methods to project these gradients into vocabulary items and explore the mechanics of how new infor- mation is stored in the LMs' neurons." sota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1357. URL https://aclanthology.org/N19-1357/. · DOI
  73. Zhuoran Jin, Pengfei Cao, Hongbang Yuan, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 1193-1215, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/ 2024.findings-acl.70. David Johnston, Arkajyoti Chakraborty, and Nora Belrose. Mechanistic anomaly detection research update, Aug 2024. URL https://blog.eleuther.ai/mad_research_update/.
  74. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2668-2677. PMLR, 10-15 Jul 2018a. URL https://proceedings.mlr.press/v80/kim18d.html.
  75. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2668-2677. PMLR, 10-15 Jul 2018b. URL https://proceedings.mlr.press/v80/kim18d.html.
  76. Louis Kirsch, Julius Kunze, and David Barber. Modular networks: learning to decompose neural compu- tation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, pp. 2414-2423, Red Hook, NY, USA, 2018.
  77. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1885-1894. PMLR, 06-11 Aug 2017. URL https://proceedings.mlr.press/v70/koh17a.html.
  78. Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5338-5348. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/koh20a.html.
  79. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024.
  80. Rudolf Laine, Bilal Chughtai, Jan Betley, Kaivalya Hariharan, Mikita Balesni, Jérémy Scheurer, Marius Hobbhahn, Alexander Meinke, and Owain Evans. Me, myself, and AI: The situational awareness dataset (SAD) for LLMs. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=UnWhcpIyUC.
  81. Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeff Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML'12, pp. 507-514, Madison, WI, USA, 2012. Omnipress. ISBN 9781450312851.
  82. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, May 2015. URL https://www.nature. com/articles/nature14539.
  83. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: a case study on dpo and toxicity. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2025.
  84. Katherine Lee, A. Feder Cooper, and James Grimmelmann. Talkin' 'bout ai generation: Copyright and the generative-ai supply chain. Journal of the Copyright Society of the USA, 2024. URL https://arxiv. org/abs/2309.08133.
  85. Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id= DeG07_TcZvT.
  86. Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Dmitry Storcheus, Afshin Rostamizadeh, and Sanjiv Kumar (eds.), Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges at NIPS 2015, volume 44 of Proceedings of Machine Learning Research, pp. 196-212, Montreal, Canada, 11 Dec 2015. PMLR. URL https://proceedings.mlr.press/v44/li15convergent.html.
  87. David Lindner, Janos Kramar, Sebastian Farquhar, Matthew Rahtz, Thomas McGrath, and Vladimir Miku- lik. Tracr: Compiled transformers as a laboratory for interpretability. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=tbbId8u7nP.
  88. Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. Rethinking machine unlearning for large language models, 2024a. URL https://arxiv.org/abs/2402. 08787.
  89. Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks, 2024c. URL https://arxiv.org/abs/ 2404.19756.
  90. Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through l 0 regu- larization. In International Conference on Learning Representations, 2018. URL https://openreview. net/forum?id=H1Y8hhg0b.
  91. Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp. 4768-4777, Red Hook, NY, USA, 2017.
  92. Aleksandar Makelov, Georg Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024. URL https://openreview.net/forum?id=MHIX9H8aYF.
  93. Alireza Makhzani and Brendan Frey. k-sparse autoencoders, 2013. URL https://arxiv.org/abs/1312. 5663.
  94. Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=aajyHYjjsk.
  95. Callum Stuart McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppres- sion: Comprehensively understanding a motif in language model attention heads. In The 7th BlackboxNLP Workshop, 2024. URL https://openreview.net/forum?id=5Hd6813x3U.
  96. Eric J Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview. net/forum?id=3tbTw2ga8K.
  97. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word repre- sentations. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff (eds.), Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pp. 746-751, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https://aclanthology.org/N13-1090.
  98. Joseph Miller, Bilal Chughtai, and William Saunders. Transformer circuit evaluation metrics are not robust. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=zSf8PJyQb2.
  99. Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=SJGCiw5gl.
  100. Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In D. Touretzky (ed.), Advances in Neural Information Processing Systems, vol- ume 1. Morgan-Kaufmann, 1988. URL https://proceedings.neurips.cc/paper_files/paper/1988/ file/07e1cd7dca89a1678042477183b7ac3f-Paper.pdf.
  101. Jesse Mu and Jacob Andreas. Compositional explanations of neurons. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17153-17163.
  102. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/ paper/2020/file/c74956ffb38ba48ed6ce977af6727275-Paper.pdf.
  103. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Repre- sentations, 2023a. URL https://openreview.net/forum?id=9XFSbDPmdW.
  104. Arunachalam Narayanaswamy, Subhashini Venugopalan, Dale R. Webster, Lily Peng, Greg S. Corrado, Paisan Ruamviboonsuk, Pinal Bavishi, Michael Brenner, Philip C. Nelson, and Avinash V. Varadarajan. Scientific discovery by generating counterfactuals using image translation. In Anne L. Martel, Purang Abolmaesumi, Danail Stoyanov, Diana Mateus, Maria A. Zuluaga, S. Kevin Zhou, Daniel Racoceanu, and Leo Joskowicz (eds.), Medical Image Computing and Computer Assisted Intervention -MICCAI 2020, pp. 273-283, Cham, 2020. Springer International Publishing. ISBN 978-3-030-59710-8.
  105. Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, pp. 3395-3403, Red Hook, NY, USA, 2016a.
  106. Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks, 2016b. URL https://arxiv.org/abs/ 1602.03616.
  107. Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks, 2016c. URL https://arxiv.org/abs/ 1602.03616.
  108. Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017b. doi: 10.23915/distill.00007. https://distill.pub/2017/feature-visualization. · DOI · Paper page
  109. Core Francisco Park, Maya Okawa, Andrew Lee, Ekdeep Singh Lubana, and Hidenori Tanaka. Emergence of hidden capabilities: Exploring learning dynamics in concept space. In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024a. URL https://openreview.net/forum?id= owuEcT6BTl.
  110. Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Causal Representation Learning Workshop at NeurIPS 2023, 2023a. URL https://openreview.net/forum?id=T0PoOJg8cK.
  111. Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. In ICML 2024 Workshop on Mechanistic Interpretability, 2024b. URL https://openreview.net/forum?id=KXuYjuBzKo.
  112. Judea Pearl. Causality. Cambridge University Press, 2 edition, 2009.
  113. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Con- ference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id= HPuSIXJaa9.
  114. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, Janos Kramar, Rohin Shah, and Neel Nanda. Improving sparse decomposition of language model activations with gated sparse autoencoders. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=zLBlin2zvW.
  115. Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept erasure. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18400-18421. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr. press/v162/ravfogel22a.html.
  116. Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context clas- sification task. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=aN4Jf6Cx69.
  117. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
  118. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openreview.net/forum?id=ITw9edRDlD.
  119. Sarah Schwettmann, Tamar Rott Shaham, Joanna Materzynska, Neil Chowdhury, Shuang Li, Jacob Andreas, David Bau, and Antonio Torralba. FIND: A function description benchmark for evaluating interpretability methods. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Bench- marks Track, 2023. URL https://openreview.net/forum?id=mkSDXjX6EM.
  120. Tamar Rott Shaham, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob An- dreas, and Antonio Torralba. A multimodal automated interpretability agent. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2025.
  121. Lee Sharkey. A technical note on bilinear layers for interpretability, 2023. URL https://arxiv.org/abs/ 2305.03452.
  122. Arnab Sen Sharma, David Atkinson, and David Bau. Locating and editing factual associations in mamba. In First Conference on Language Modeling, 2024a. URL https://openreview.net/forum?id=yoVRyrEgix.
  123. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations, 2024b. URL https://openreview.net/forum?id=tvhaxkMKAn.
  124. Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=B1ckMDqlg.
  125. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014b. URL https://arxiv.org/abs/1312.6034. · arXiv
  126. Dylan Slack, Anna Hilgard, Himabindu Lakkaraju, and Sameer Singh. Counterfactual explanations can be manipulated. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 62-75. Cur- ran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ 009c434cab57de48a31f6b669e7ba266-Paper.pdf.
  127. P. Smolensky. Neural and conceptual interpretation of PDP models, pp. 390-431. MIT Press, Cambridge, MA, USA, 1986. ISBN 0262631105.
  128. Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learn- ing using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Re- search, pp. 2256-2265, Lille, France, 07-09 Jul 2015. PMLR. URL https://proceedings.mlr.press/ v37/sohl-dickstein15.html.
  129. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56): 1929-1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html.
  130. Dashiell Stander, Qinan Yu, Honglu Fan, and Stella Biderman. Grokking group multiplication with cosets. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2025.
  131. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, ICML'17, pp. 3319-3328. JMLR.org, 2017.
  132. Hannes Thurnherr and Jérémy Scheurer. Tracrbench: Generating interpretability testbeds with large lan- guage models. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. URL https://openreview. net/forum?id=vNubZ5zK8h.
  133. Shengbang Tong, Erik Jones, and Jacob Steinhardt. Mass-producing failures of multimodal systems with language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024a.
  134. Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 74952-74965.
  135. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/ paper/2023/file/ed3fea9033a80fea1376299fa7863f4a-Paper-Conference.pdf.
  136. Guillermo Valle-Perez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rye4g3AqFm.
  137. Teun van der Weij, Felix Hofstätter, Ollie Jaffe, Samuel F. Brown, and Francis Rhys Ward. Ai sandbagging: Language models can strategically underperform on evaluations, 2024. URL https://arxiv.org/abs/ 2406.07358.
  138. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Sys- tems, volume 33, pp. 12388-12401.
  139. Curran Associates, Inc., 2020. URL https://proceedings.neurips. cc/paper_files/paper/2020/file/92650b2e92217715fe312e6fa7b90d82-Paper.pdf.
  140. Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Asso- ciation for Computational Linguistics ACL 2024, pp. 1288-1301, Bangkok, Thailand and virtual meet- ing, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024. findings-acl.75.
  141. Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization. In ICML 2024 Workshop on Mechanistic Interpretability, 2024a. URL https://openreview.net/forum?id=ns8IH5Sn5y.
  142. George Wang, Matthew Farrugia-Roberts, Jesse Hoogland, Liam Carroll, Susan Wei, and Daniel Murfet. Loss landscape geometry reveals stagewise development of transformers. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024b. URL https://openreview.net/ forum?id=2JabyZjM5H.
  143. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretabil- ity in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=NpsVSN6o4ul.
  144. Francis Ward, Francesca Toni, Francesco Belardinelli, and Tom Everitt. Honesty is the best policy: Defin- ing and mitigating ai deception. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 2313-2341.
  145. Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2009.
  146. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Ma- chine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=yzkSU5zdwD. Survey Certification.
  147. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024.
  148. Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11080-11090. PMLR, 18-24 Jul 2021. URL https://proceedings. mlr.press/v139/weiss21a.html.
  149. Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=nRfClnMhVX.
  150. Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. ReFT: Representation finetuning for language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https://openreview.net/forum? id=fykjplMc0V.
  151. Rui Xin, Chudi Zhong, Zhi Chen, Takuya Takagi, Margo Seltzer, and Cynthia Rudin. Exploring the whole rashomon set of sparse decision trees. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/ forum?id=WHqVVk3UHr.
  152. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30378-30392.
  153. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ c402501846f9fe03e2cac015b3f0e6b1-Paper-Conference.pdf.
  154. Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision -ECCV 2014, pp. 818-833, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10590-1.
  155. Xinyang Zhang, Ningfei Wang, Hua Shen, Shouling Ji, Xiapu Luo, and Ting Wang. Interpretable deep learning under fire. In Proceedings of the USENIX Security Symposium (Security), 2020.
  156. Roland Simon Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas S. A. Wallis, and Wieland Brendel. How well do feature visualizations support causal understanding of CNN activations? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=vLPqnPf9k0.
  157. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023b. URL https://arxiv.org/abs/2307. 15043.

Cited by

Sorted by citations

See all 291

Related

See all

Metadata

Authors
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., Biderman, S., Garriga-Alonso, A., Conmy, A., Nanda, N., Rumbelow, J., Wattenberg, M., Schoots, N., Miller, J., Michaud, E. J., Casper, S., Tegmark, M., Saunders, W., Bau, D., Todd, E., Geiger, A., Geva, M., Hoogland, J., Murfet, D., & McGrath, T.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2501.16496
arXiv
2501.16496
OpenAlex
W4406949968
Semantic Scholar
8a94d7fb8b580621979396042aef89dbd6ec37fb
Cited by
291
References
197
Citations by year
2026: 163 · 2025: 123
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
mechanistic interpretability, artificial neural networks, nature of intelligence, computational mechanisms, socio-technical challenges, open problems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review