Open Problems in Mechanistic Interpretability

In the paper

“Other studies leverage architectures that are mathematically analyzable in other ways than linearity, such bilinear activations in MLPs (Sharkey, 2023;Pearce et al., 2024).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Sharkey, 2023
• Supported

  The Introduction (§1) says bilinear MLP computations can be expressed using linear operations and a third-order tensor, enabling tensor or matrix decomposition of their weights. It also describes the paper’s own method for decomposing bilinear MLPs into interpretable features, supporting the claim that the study leverages this mathematically analyzable architecture. (LLM, conf=0.93)

  cites Pearce et al., 2024 source

In the paper

“Probes were first introduced in NLP (Köhn, 2015;Gupta et al., 2015) and have since been extensively explored in the field (Conneau et al., 2018;Tenney et al., 2019;Rogers et al., 2020;Gurnee et al., 2023;Peters et al., 2018;Burns et al., 2023;Marks & Tegmark, 2024).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Köhn, 2015
• Partly supported

  Section 2.3 (Attribute Prediction; page not provided) describes training logistic-regression models on word-vector dimensions to predict referential attributes, an early example of probing representations in NLP. The paper does not establish that probes were first introduced in NLP by this work. The sentence’s claim about subsequent extensive exploration is attached to other citations, not this one. (LLM, conf=0.84)

  cites Gupta et al., 2015 sourceExperimental — accuracy not yet measured
• Supported

  The citation to Conneau et al. (2018) is attached to the claim that probes have been extensively explored in NLP, not to the claim about where they were first introduced. The Introduction describes ten probing tasks and their use across a wide range of encoding architectures and training methods; the Related work passage also identifies earlier probing studies. (LLM, conf=0.90)

  cites Conneau et al., 2018 source

In the paper

“However, attribution patching uses gradients, which only yield a first-order approximation of the effect of ablating components (Wu et al., 2024c;Molchanov et al., 2017), leaving it unclear whether this method and any improvements on it (Kramár et al., 2024) produce adequate approximations.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Wu et al., 2024c
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Molchanov et al., 2017
• Supported

  Section 2.2 defines attribution patching as a first-order Taylor approximation using a gradient. The Abstract identifies AtP* as an improved variant, while the Conclusion notes remaining failure modes and recommends reliability diagnostics. Together, these passages support the citing sentence’s description of the method and its improvements as approximations whose adequacy requires scrutiny. (LLM, conf=0.90)

  cites Kramár et al., 2024 source

In the paper

“For instance, Hilton et al. (2020) validated their methods by explaining cases where a deep reinforcement learning agent's neural network failed to achieve maximum reward and also explained specific hallucinations exhibited by the network.”

What the sources say

• Partly supported

  The “Dissecting failure” and “Hallucinations” sections describe explaining failures to achieve maximum reward and specific mistaken detections, including a background shape mistaken for a coin. But the “Model editing” section says the authors quantitatively validated their analysis by editing the model and testing the effects, not by explaining those cases. (LLM, conf=0.86)

  cites Hilton et al., 2020 sourceExperimental — accuracy not yet measured

In the paper

“Meanwhile, Marks et al. (2024) showed how an interpretability tool could assist humans with debugging a classifier in a toy task.”

What the sources say

• Supported

  Section 4 describes SHIFT, in which a human inspects an interpretable feature circuit and removes features judged irrelevant to the classifier’s task. The authors demonstrate it on a simplified profession-classification task where gender is a misleading signal, and report that the intervention substantially reduces the classifiers’ dependence on gender. (LLM, conf=0.91)

  cites Marks et al., 2024 source

In the paper

“When attempting to localize a human-interpretable concept within the network, an intuitive approach is to 'probe' for it (Köhn, 2015;Gupta et al., 2015;Alain & Bengio, 2017;Ettinger et al., 2016).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Köhn, 2015
• Partly supported

  The Introduction describes training a regression model on distributional vectors to retrieve interpretable attributes of countries and cities, which is an example of probing a representation. The paper does not attempt to localize a concept within a network; it tests what can be recovered from word vectors. (LLM, conf=0.83)

  cites Gupta et al., 2015 sourceExperimental — accuracy not yet measured
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Alain & Bengio, 2017

In the paper

“Attribution patching Syed et al. (2024) was designed to identify potential relevant candidates for further testing through intervention, which becomes more important as the number of components expands significantly through sparse dictionary learning (Marks et al., 2024).”

What the sources say

• Supported

  Section 3 describes Edge Attribution Patching as a technique for identifying relevant model components. Section 5.2 reports using it to prune a subgraph before applying ACDC, which tests edges through activation-patching interventions; the Conclusion recommends that sequence for future circuit discovery. (LLM, conf=0.90)

  cites Syed et al., 2024 source
• Supported

  The Marks et al. citation supports the second clause, about the importance of efficient methods when sparse dictionary learning expands the number of components; the attribution-patching design claim is cited to Syed et al. In §2 (Formulation), Marks et al. report sparse-autoencoder feature dimensions 64 and 8 times the original latent dimension and explain that attribution patching estimates effects for many components in parallel. Our copy provides no page number for these passages. (LLM, conf=0.90)

  cites Marks et al., 2024 source

In the paper

“Probing also includes concept activation vectors (Kim et al., 2018a), information-theoretic probing (Voita & Titov, 2020), and structural probing (Hewitt & Manning, 2019).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Kim et al., 2018a
• Supported

  Section 2, “Information-Theoretic Viewpoint” (page not provided), explicitly proposes examining probing from an information-theoretic perspective. The Voita and Titov citation supports the information-theoretic probing item in the list; the other items have separate citations. (LLM, conf=0.93)

  cites Voita & Titov, 2020 source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Hewitt & Manning, 2019

In the paper

“However, more recent work has also used SDL latents for nodes (Marks et al., 2024)”

What the sources say

• Supported

  In “Viewing SAE features as part of the model,” the paper says its computation-graph nodes correspond to SAE feature activations or SAE errors at particular token positions. The Formulation section describes SAEs as identifying sparse, interpretable latents, supporting the paraphrase that the work uses such latents as nodes. (LLM, conf=0.93)

  cites Marks et al., 2024 source

In the paper

“However, training a probe to associate a concept with specific hidden activations does not necessarily imply that those activations causally mediate how that concept is used by the network, or even if the network uses the concept at all (Ravichander et al., 2021;Geiger et al., 2024b;Elazar et al., 2021;Belinkov, 2022b).”

What the sources say

• Supported

  The Discussion states that probing cannot determine whether a property is needed for a task or pinpoint the information a model relies on. It also distinguishes probing from causal methods that intervene in representations to test their effect on predictions. Together, these passages support both parts of the paraphrase. (LLM, conf=0.93)

  cites Ravichander et al., 2021 source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Geiger et al., 2024b
• Supported

  The Introduction (p. 1) says that probing can show information is extractable from a representation but provides no evidence that the model uses it. Section 4 (p. 4) reports properties detectable by probing that the model does not use for prediction, supporting the warning against inferring a causal role from probe performance. (LLM, conf=0.94)

  cites Elazar et al., 2021 source

In the paper

“Farrell et al. (2024) use unlearning (Section 3.2.2) to demonstrate that learned SDL latents don't quite match human concepts, and might not be optimal for particular downstream use cases, highlighting potential issues with SDL.”

What the sources say

• Partly supported

  Section 5 (Conclusion; page not provided) reports that SAE-based unlearning has similar or greater unwanted side effects than a comparison method and says that SAE training or the intervention method needs substantial improvement. This supports concerns about SAEs for the downstream task of unlearning, but the paper does not demonstrate that learned latents fail to match human concepts; Section 3.1 instead describes a representative feature as relatively monosemantic. (LLM, conf=0.80)

  cites Farrell et al., 2024 sourceExperimental — accuracy not yet measured

In the paper

“Conflating hypotheses with conclusions has regrettably been commonplace in mechanistic interpretability research, making validation an important area for the field to improve (Madsen et al., 2024;Stander et al., 2025).”

What the sources say

• Partly supported

  Section 1.1 says that validating whether explanations faithfully reflect a model is challenging and warns against taking faithfulness for granted; the Conclusion calls for measurable definitions of faithfulness. But Section 2.2.2 merely describes mechanistic interpretability. The paper does not say that conflating hypotheses with conclusions is commonplace in mechanistic interpretability research. (LLM, conf=0.84)

  cites Madsen et al., 2024 sourceExperimental — accuracy not yet measured
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Stander et al., 2025

In the paper

“Circuit discovery methods are generally based on iterative activation patching (Wang et al., 2023;Chan et al., 2022a;Lieberum et al., 2023) or integrated gradients (Marks et al., 2024).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Wang et al., 2023
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Chan et al., 2022a
• Supported

  The Formulation section describes an integrated-gradients-based approximation for estimating the effects used in circuit discovery. This supports citing Marks et al. as an example of an integrated-gradients-based method; the other citations accompany the activation-patching part of the sentence. (LLM, conf=0.90)

  cites Marks et al., 2024 source

In the paper

“They have also been applied in vision (Alain & Bengio, 2017;Kim et al., 2018b) and deep reinforcement learning (McGrath et al., 2022;Forde et al., 2023).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Alain & Bengio, 2017
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Kim et al., 2018b
• Supported

  McGrath et al. is cited for the deep-reinforcement-learning part of the sentence, not the vision part. In “Probing for Human Concepts,” the authors describe training linear probes on AlphaZero’s network activations; “AlphaZero Training Iterations” describes the network learning from self-play. Together, these passages support the citation as an example of probes applied in deep reinforcement learning. (LLM, conf=0.93)

  cites McGrath et al., 2022 source

In the paper

“Adding a trainable affine or linear transformation before unembedding, as also referred to as the tuned lens, improves decoding accuracy (Belrose et al., 2023a;Yom Din et al., 2024), at the cost of less faithfully representing when the model has completed computation.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Belrose et al., 2023a
• Partly supported

  The Introduction reports that the paper’s learned linear mappings improved prediction-estimation accuracy over naive projections. Related Work distinguishes those mappings from Belrose et al.’s affine tuned-lens method, and the cited passages do not establish that greater accuracy comes at the cost of less faithfully showing when computation is complete. (LLM, conf=0.86)

  cites Yom Din et al., 2024 sourceExperimental — accuracy not yet measured

In the paper

“Other work learns masks over network components to remove irrelevant components (De Cao et al., 2020;Csordás et al., 2021;Davies et al., 2023).”

What the sources say

• Supported

  The Conclusion states that DIFFMASK learns to remove subsets of inputs or hidden states through masking. The Introduction explains that its learned gates disregard inputs whose information need not be propagated, supporting the description of removing irrelevant components. (LLM, conf=0.91)

  cites De Cao et al., 2020 source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Csordás et al., 2021
• Partly supported

  Section 3, “Using Desiderata to Localize Computation” (page unavailable), says the authors learn a sparse mask over model components to identify circuitry for a task. It defines the mask as controlling which components are patched or left unchanged, not as removing irrelevant components from the network. (LLM, conf=0.86)

  cites Davies et al., 2023 sourceExperimental — accuracy not yet measured

In the paper

“Although relatively simple to implement, probing has two main challenges (Ravichander et al., 2021;Belinkov, 2022b): (1) The need for carefully chosen data for well-defined concepts, and (2) probes detect correlations instead of causal variables in hidden activations.”

What the sources say

• Supported

  The Discussion (§6) says datasets may imperfectly reflect task requirements and calls for careful identification of the competencies they represent. It also says probing can detect a correlated property in a representation without establishing that the model uses it for the task, and points to causal interventions as an alternative. These passages support the two challenges in the citing sentence, though the paper does not label them an exhaustive list of two. (LLM, conf=0.86)

  cites Ravichander et al., 2021 source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Belinkov, 2022b

In the paper

“Currently, the techniques used for this purpose are mostly unrelated to interpretability (e.g.Christiano et al. (2017); Rafailov et al. (2023); inter alia), but sometimes inspired by it (Rimsky et al. (2024), Zou et al. (2024), Kirch et al. (2024); inter alia).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Christiano et al. (2017)
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Rafailov et al. (2023)
• Supported

  The Rimsky citation is attached to the claim that some behavior-control techniques are inspired by interpretability, not to the claim that most are unrelated to it. The Introduction describes CAA’s approach as steering behavior through targeted changes to model activations, and the Discussion says it also yields insight into the model’s internal representations. That supports citing CAA as an example of the interpretability-related kind of technique. (LLM, conf=0.87)

  cites Rimsky et al. (2024) source

In the paper

“For example, prior work identified signatures in model internals that predict a model's likelihood of hallucinating (Yu et al., 2024) or its knowledge of particular facts (Gottesman & Geva, 2024).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Yu et al., 2024
• Partly supported

  The Introduction and §2 describe probes of internal representations that estimate how knowledgeable a model is about a given entity, including its expected accuracy on questions about that entity. They do not establish that the probes predict whether the model knows a particular fact; the paper distinguishes its entity-level estimate from predicting the factuality of a specific claim. The hallucination-detection portion of the citing sentence is attributed to Yu et al., not this source. (LLM, conf=0.85)

  cites Gottesman & Geva, 2024 sourceExperimental — accuracy not yet measured

In the paper

“In interpretable AI research, the motivations and methods employed are often described as”

What the sources say

• Supported

  The Introduction (p. 1) says that both the motives for interpretability and the technical descriptions of interpretable models are “diverse and occasionally discordant.” This supports the citing sentence’s paraphrase about motivations and methods in interpretable AI research. (LLM, conf=0.94)

  cites Lipton, 2018 source

In the paper

“Prakash et al. (2024) find evidence that finetuning improves capabilities primarily by enhancing existing circuits, rather than developing fundamentally new mechanisms.”

What the sources say

• Partly supported

  The Introduction (§1) says the findings indicate that fine-tuning enhanced an existing mechanism rather than causing a fundamental shift, supporting the core paraphrase. But the Discussion (§7) limits the evidence to entity tracking in three fine-tuned models and says further tasks and models are needed to know whether this pattern is typical. The claim's unqualified reference to capabilities is broader than the study establishes. (LLM, conf=0.85)

  cites Prakash et al., 2024 sourceExperimental — accuracy not yet measured

In the paper

“Failing to do so or misrepresenting these comparisons can lead to follow-up work that rests on false assumptions (Lipton & Steinhardt, 2019;Leech et al., 2024).”

What the sources say

• Supported

  Section 3.2 (p. 3) says inadequate evaluation can mislead readers about which changes produced empirical gains, and describes purported improvements overturned by simpler baselines. Section 6 (p. 9) says such flawed scholarship negatively affects the research community, supporting the claim that misleading comparisons can leave subsequent work resting on false assumptions. (LLM, conf=0.86)

  cites Lipton & Steinhardt, 2019 source
• Partly supported

  Sections 3.2.1 and 3.3.7 describe weak or undertuned comparators and conclusions that exaggerate evaluation results, supporting the concern about misleading comparisons. Our copy does not explicitly say that such comparisons cause follow-up work to rest on false assumptions. (LLM, conf=0.82)

  cites Leech et al., 2024 sourceExperimental — accuracy not yet measured

In the paper

“Another potential reason for lack of clarity is that models are often studied in a vacuum: Smart & Kasirzadeh (2024) emphasize that the usefulness or correctness of model interpretations can depend on the broader context of the model's development or deployment.”

What the sources say

• Supported

  Section 2 says ML models do not operate in isolation: social and institutional contexts can influence their behavior, and socio-structural factors can improve the soundness of explanations. Section 5 likewise argues that model-centric explanations can be inadequate for understanding models developed and deployed in complex social contexts. The cited copy provides no page numbers. (LLM, conf=0.93)

  cites Smart & Kasirzadeh, 2024 source

In the paper

“For example, there is work removing the layer norm operations (Heimersheim, 2024), using input-switched affine transformations for recurrence (Foerster et al., 2017).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Heimersheim, 2024
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Foerster et al., 2017

In the paper

“Given the practical and conceptual issues with SDL, there is broad agreement that the question of how to correctly decompose networks into atomic units remains a central problem, evidenced by the large amount of effort focused on the direction in recent years.”

In the paper

“Moreover, although there is much emphasis on how models represent features in superposition, there is comparatively less work examining how they might perform computation on them natively in superposition.”

In the paper

“One such assumption is the linear representation hypothesis.”

In the paper

“Attribution methods are necessary for causal explanations but are often difficult to interpret.”

In the paper

“However, criticisms of feature synthesis methods suggest that natural dataset examples may serve interpretation better (Zimmermann et al., 2021;Borowski et al., 2021) or show that current methods struggle to identify trojans (Casper et al., 2023a).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Zimmermann et al., 2021
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Borowski et al., 2021
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Casper et al., 2023a

In the paper

“Han et al. (2015) achieve sparse weights by using magnitude pruning followed by finetuning.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Han et al., 2015

In the paper

“Currently, SDL is the most popular set of unsupervised decomposition methods in mechanistic interpretability.”

In the paper

“Without solid conceptual foundations, it remains unclear whether the superposition hypothesis, which underpins the SDL paradigm, is fundamentally valid or merely pragmatically useful (Henighan, 2024;Templeton et al., 2024).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Henighan, 2024
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Templeton et al., 2024

In the paper

“Existing research implements other approaches that can simplify linearity-reliant circuit analysis (Elhage et al., 2021).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Elhage et al., 2021

In the paper

“After decomposing a network into components, whether through post hoc decomposition or by using intrinsically decomposable models, our task remains unfinished.”

In the paper

“After investing considerable effort in SDL approaches, it is apparent that improved conceptual clarity beyond the idea of superposition (Elhage et al., 2021) is needed to advance neural network decomposition.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Elhage et al., 2021

In the paper

“Similarly, Makelov et al. (2024) found that using reconstructions from sparse autoencoders decreased GPT-2 small performance by 10% when trained on task-specific data, and 40% when trained on the full distribution.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Makelov et al., 2024

In the paper

“However, it is not immediately obvious how to decompose representations distributed across attention heads with SDL (Mathwin et al., 2024;Wynroe & Sharkey, 2024).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Mathwin et al., 2024
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Wynroe & Sharkey, 2024

In the paper

“The linear representation hypothesis observes that though neural networks are nonlinear functions and could potentially use highly nonlinear representations (Black et al., 2022;Engels et al., 2024b;Kirch et al., 2024), they tend to use representations that exhibit strikingly linear behavior (Smolensky, 1986;Mikolov et al., 2013;Olah et al., 2020b;Elhage et al., 2021;Park et al., 2023a;Guerner et a”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Smolensky, 1986
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Mikolov et al., 2013
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Elhage et al., 2021

In the paper

“According to the 'superposition hypothesis', neural networks are capable of representing more features than they have dimensions, as long as each feature activates sparsely (Elhage et al., 2021) (Figure 3).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Elhage et al., 2021

In the paper

“Developing efficient and accurate attribution methods thus remains an open problem.”

In the paper

“Some work attempts to ground mechanistic interpretability in the formalisms of causality (Geiger et al., 2024a).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Geiger et al., 2024a

In the paper

“This means that plausible explanations based on highly activating data set examples cannot be solely relied upon to identify the basic units of computations in neural networks -other methods are needed to accurately identify them.”

In the paper

“However, even without foundations, progress in mechanistic interpretability is possible -even confused concepts can be pragmatically useful (Henighan, 2024).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Henighan, 2024

In the paper

“(Kissane et al., 2024c) found that SAEs trained on pretraining data generally do not have good latents for the concept of 'refusing' harmful user requests, while SAEs trained on chat formatted data do.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Kissane et al., 2024c

In the paper

“Interpretability researchers have historically faced challenges in adequately validating their hypotheses due to the high costs in terms of time and cognitive labor.”

In the paper

“Hypotheses in interpretability require extensive validation beyond what appearances might imply.”

In the paper

“If one description method yields a different description from another, it invalidates the hypothesis, which necessitates returning to an earlier step in the reverse engineering cycle (Figure 2).”

In the paper

“Moreover, it is only possible to identify concepts that we have defined precisely enough to create high-quality data.”

In the paper

“While these methods can identify causal mediators of concepts in hidden representations, they require more specialized data than probes do.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Geiger et al., 2024c
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Wu et al., 2023
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Huang et al., 2024a

In the paper

“A causal intervention typically requires a forward pass of the model.This may make performing one for every network component in large models, long contexts, or when using finer-grained components such as sparse autoencoder latents prohibitively expensive.”

In the paper

“With the increasing sophistication of language models, researchers have generated descriptions of the functional role of neurons in image models (Hernandez et al., 2022), neurons in language models (Bills et al., 2023), and sparse autoencoder latents in language models (Huben et al., 2024;Bricken, 2023;Juang et al., 2024) using highly activating data set examples.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Hernandez et al., 2022
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Bills et al., 2023
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Huben et al., 2024

In the paper

“Testing the hypothesis often requires multiple approaches (Mueller et al., 2024).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Mueller et al., 2024

In the paper

“Benchmarking is a proven approach to making incremental improvements in other areas of machine learning, with several approaches to benchmarking interpretability methods being developed in recent years.”

In the paper

“Although interpretability is sometimes motivated by achieving engineering goals, it is often also approached through the perspective of the natural sciences (Olah et al., 2020b).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Olah et al., 2020b

In the paper

“Currently, interpretability researchers lack consensus on which networks should serve as model organisms.”

In the paper

“Circuit discovery has emerged as a prominent pipeline in recent mechanistic interpretability research (Wang et al., 2023;Hanna et al., 2023;Heimersheim & Janiak, 2023).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Wang et al., 2023
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Hanna et al., 2023
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Heimersheim & Janiak, 2023

In the paper

“Studying solely model organisms, instead of more directly pursuing engineering goals, risks merely making true statements about neural network structure, rather generating insights that are of immediate practical benefit.”

In the paper

“In research papers, this loop is rarely made explicit, as only the final description is presented.”

In the paper

“Miller et al. (2024) demonstrate that the within-task variance of model performance across the distribution of data points in a task is large, implying that the circuit provides a good approximation of the average case performance on the dataset, but a poor one for any individual data point.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Miller et al., 2024

In the paper

“Other interpretability benchmarks that evaluate specific steps in the interpretability pipelines also exist, such as model decomposition (Huang et al., 2024a;Makelov et al., 2024), generating descriptions of network component functions (Schwettmann et al., 2023) , or testing natural language explanations (Huang et al., 2023).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Huang et al., 2024a
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Makelov et al., 2024
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Schwettmann et al., 2023

In the paper

“This is a concept-based step, since the definition of the task was based on how human researchers define a task distribution.”

In the paper

“Circuits are commonly evaluated based on three attributes (Wang et al., 2023): faithfulness, which refers to how closely the circuit approximates the entire network's behavior, minimality, which assesses if nodes in the subgraph are unnecessary, and completeness, which determines whether any nodes not included in the subgraph are important for task behavior.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Wang et al., 2023

In the paper

“The use of model organisms allows for cross-checking results with previous work, enabling stronger validation of hypotheses.”

In the paper

“However, limiting its focus by studying small toy models (e.g.Nanda et al. ( 2023a)) or how larger models accomplish select subtasks (Arditi et al., 2024), risks incentivizing research and methods that fail to generalize to more safety-relevant real-world settings.”

In the paper

“For example, Templeton et al. (2024) discussed examples where manually editing a large language model based on an interpretation led to predictable high-level changes in its behavior.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Templeton et al., 2024

In the paper

“For instance, distributed alignment search (Geiger et al., 2024c) learns a linear subspace that represents a particular concept, using data with interventions on that concept as supervision (Geiger et al., 2024c;Guerner et al., 2024;Wu et al., 2023).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Geiger et al., 2024c
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Guerner et al., 2024
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Wu et al., 2023

In the paper

“Obtaining a labeling function is not always trivial and may require substantial human effort to define a data set for a single concept.”

In the paper

“Historically, mechanistic interpretability research has required considerable manual researcher effort, though it typically studies models that are smaller than those at the frontier.”

In the paper

“The success of these predictions can be used as a quantitative measurement of 'interpretability'.”

In the paper

“Model organisms not only support cross-validation of hypotheses, but also facilitate the progressive construction of experimental infrastructure by providing a reliable foundation for experiment design.”

In the paper

“Confirming such hypotheses requires further investigation using causal interventions or other probing methods.”

In the paper

“Causal scrubbing (Chan et al., 2022a) is a generalization of path patching that allows for testing hypotheses concerning any given connection between a set of network components.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Chan et al., 2022a

In the paper

“If the probe is a linear model, then we have localized the concept as a vector in latent space.”

In the paper

“Most work thus far has selected architectural components (Section 2.1.2),such as attention heads and MLP layers, to be the nodes.”

In the paper

“Cloud et al. (2024) incentivize modularity via applying data-dependent, weighted masks to gradients during backpropagation.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Cloud et al., 2024

In the paper

“Decomposition vs. description of network components: Improvements in network decomposition versus component description methods offer varying benefits for different goals.”

In the paper

“For a strong version of this claim, substantial progress in both decomposition and description methods is necessary.”

In the paper

“It may not be necessary to have deep descriptions of the function of individual mechanisms as long as anomalies can be detected.”

In the paper

“The overwhelming majority of mechanistic interpretability research focuses on the transformer model family and therefore does not distinguish between observations that are model-specific and those that are not.”

In the paper

“At times, the objective of AI interpretability research is articulated as advancing a fundamental”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Christensen & Cheney, 2015

In the paper

“The most competitive methods of AI development systems result in uninterpretable systems that often fail in ways that surprise their developers (OpenAI et al., 2024;Team et al., 2024;Anthropic, 2024).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites OpenAI et al., 2024
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Team et al., 2024
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Anthropic, 2024

In the paper

“Mechanistic anomaly detection primarily requires progress in decomposition methods (Section 2.1.2) as it is necessary to be confident about what constitutes an individual mechanism within the network.”

In the paper

“Editing any given capability or piece of knowledge presents a greater challenge than deleting them.”

In the paper

“Current methods like sparse dictionary learning, while promising, face both practical limitations in scaling to larger models and deeper conceptual challenges regarding their underlying assumptions.”

In the paper

“The extent of network decomposition or description needed may vary depending on the goal.”

In the paper

“Current SDL methods identify active latents, but not active mechanisms, which are implemented by network parameters.”

In the paper

“The majority of other areas of machine learning already focus on controlling or steering the behavior of AI systems to alter input-output behavior.”

In the paper

“Meaningful progress in unlearning and editing methods may depend on improved network decomposition methods, as it would require isolating the individual mechanisms that correspond to specific knowledge or capabilities.”

In the paper

“Knowledge and capabilities could, in fact, be part of large mechanisms that overlap with each other, making it challenging to isolate them into discrete components.”

In the paper

“Further methodological progress is required for fully automated neural network interpretability to be capable of generating the quality of interpretations necessary to achieve our goals.”

In the paper

“Accurately predicting model behavior in new scenarios or regimes is difficult (arguably impossible) without understanding model internals.”

In the paper

“Understanding their values or goals (or, less anthropomorphically, 'the internal mechanisms that determine their action plans and actions') should enable us to better predict their behavior across a broad range of contexts (Colognese & Jose, 2023).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Colognese & Jose, 2023

In the paper

“The absence of paradigmatic clarity is a major socio-technical factor for this.”

In the paper

“The scope of task distribution analysis also depends on the intended goal.”

In the paper

“However, weaker versions of the claim, addressing specific undesirable behaviors, might be more feasible with near-term methods.”

In the paper

“To be useful, it may not even be necessary to decompose or describe the entire network; having descriptions for components that are used on concerning subdistributions of model behavior might suffice.”

In the paper

“For mechanistic interpretability to effectively guide editing, strong description methods will be necessary to understand how to modify specific targets without affecting others.”

In the paper

“Mechanistic understanding post vs. during training: Understanding the mechanisms of a fixed model might suffice for some goals, but more ambitious goals might require an understanding of not only the models' mechanisms, but also how they change during the learning process.”

In the paper

“Turner et al. (2024) introduced activation steering, directly inspired by the Linear Representation Hypothesis (discussed in Paragraph 2.1.2c).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Turner et al., 2024

In the paper

“A model's behavior, which may only become apparent in unforeseen circumstances, cannot be fully captured by its performance on a finite set of behavioral evaluations.”

In the paper

“Other work shows a correlation between in-context learning capabilities and the emergence of induction heads, an attention-based circuit mechanism (Olsson et al., 2022).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Olsson et al., 2022

In the paper

“Deeper descriptions of a component are typically more causal or mechanistic, whereas shallower descriptions may rely more on correlations and only connect to inputs or outputs without referencing intermediate causes or effects (Figure 5.”

In the paper

“Recent studies show that certain methods transfer from CNNs to SSMs (Paulo et al., 2024), and from transformers to some SSMs e.g. ( Meng et al. (2022b)”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Paulo et al., 2024
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Meng et al. (2022b)

In the paper

“For example, OpenAI's Preparedness Framework commits to mitigating biological risks posed by its AI systems (OpenAI, 2023), but the framework lacks details on specific measures that might be taken.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites OpenAI, 2023

In the paper

“On the other hand, interpretability is perhaps the only research area that attempts to understand the mechanisms of model cognition.”

In the paper

“Note that ACDC-like approaches in general assist in identifying relevant subgraphs for a pre-defined task, but do not automate important subsequent steps, such as describing the functional role of subgraph components.”

In the paper

“Further advancement in tools for comparing feature-level differences between models (such as Lindsey et al. ( 2024)) may accelerate our ability to debug finetuning or other control methods (Bricken et al., 2024).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Lindsey et al. (2024)
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Bricken et al., 2024

In the paper

“Each axis of methodological advancement leads to varying degrees of progress toward different goals.”

In the paper

“The same data may lead to different conclusions under different definitions of fairness.”

In the paper

“Meanwhile, recent work proposes incorporating SDL (Paragraph 2.1.2c)into safety cases for advanced AI systems.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Grosse, 2024