Open Problems in Mechanistic Interpretability

The strong version of the linear representation hypothesis is false for some models.

“Some works have shown that the strong version is false for some models (Black et al., 2022;Csordás et al., 2024).”

• cites Black et al., 2022 sourceSupported
• cites Csordás et al., 2024 sourceNot checked yet

Bias in data sets can create misleading explanations even when top activating examples are selected from real data sets, as opposed to synthetically created data.

“Bolukbasi et al. (2021) show that bias in data sets can create misleading explanations even when top activating examples are selected from real data sets, as opposed to synthetically created data.”

• cites Bolukbasi et al., 2021 sourceSupported

The highly activating data set examples approach has the potential for 'interpretability illusions'.

“Another issue with this approach is the potential for 'interpretability illusions'.”

• cites Bolukbasi et al., 2021 sourceSupported

Humans and AI models often solve problems in different ways.

“However, humans and AI models often solve problems in different ways.”

• cites Shlegeris et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Kosoy et al., 2023 sourceNot checked yet
• cites Jumper et al., 2021 sourceCould not verify

  The Discussion describes how AlphaFold predicts protein structures, and Related work compares computational approaches. Neither section compares how humans and AI models solve problems. Searches for human experts and differing problem-solving approaches found no passage addressing the claim; page numbers were not available in our copy. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

An adversary can train a model or perturb an input to reveal any attribution map.

“Furthermore, an adversary can train a model or perturb an input to reveal any attribution map (Dombrowski et al., 2019;Ghorbani et al., 2019;Heo et al., 2019;Kindermans et al., 2019;Slack et al., 2020;Zhang et al., 2020) foot_3 .”

• cites Dombrowski et al., 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ghorbani et al., 2019 sourcePartly supported

  The Abstract (p. 1) and Methods (p. 4) show that adversarial input perturbations can dramatically change feature-importance maps, including directing importance toward a predefined region. They do not establish that an adversary can reveal any arbitrary attribution map, and the cited attacks perturb inputs rather than train a model. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites Heo et al., 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Studying the attention patterns of models can often be misleading.

“More broadly, research suggests that studying the attention patterns of models can often be misleading (Jain & Wallace, 2019;Pruthi et al., 2020).”

• cites Jain & Wallace, 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Pruthi et al., 2020 sourceSupported

With a large enough number of experts and sufficient activation sparsity, experts in mixture-of-experts models may become individually interpretable.

“Several approaches, such as 'mixture of experts' (Shazeer et al., 2017;Fedus et al., 2022;He, 2024), use sparsely activating components -with a large enough number of experts, and with sufficient activation sparsity, experts may become individually interpretable (Warstadt et al., 2019).”

• cites Warstadt et al., 2019 sourceCould not verify

  The Experiments and Discussion sections address neural-network models for sentence acceptability judgments, not mixture-of-experts models. Searches for activation sparsity and individually interpretable experts found no passage addressing the attributed claim; our copy therefore neither supports nor contradicts it. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Error nodes contain 'everything else', making them an inadequate solution to the reconstruction error problem.

“However, while the sparse autoencoders identified interpretable latents, the error nodes contain 'everything else', making them an inadequate solution to the problem.”

• cites Marks et al., 2024 sourcePartly supported

  Section 2 (Formulation; page unavailable) defines SAE error terms as components not captured by interpretable features, and Section 8 (Limitations; page unavailable) says components not captured by SAEs remain uninterpretable. That supports the concern behind 'everything else,' but the paper describes incorporating error nodes as a principled way to account for them; it does not call them an inadequate solution to reconstruction error. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

The highly activating data set examples method relies on human prior beliefs, which may lead interpreters to project their human understanding onto models that may be using unfamiliar concepts.

“The first issue is that the method relies on human prior beliefs, which may lead interpreters to project their human understanding onto models that may, in fact, be using unfamiliar concepts.”

• cites Freiesleben & König, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Donnelly & Roegiest, 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Gale et al., 2020 sourcePartly supported

  Section 3.12 describes asking people to identify a common object in image panels, and Section 4.2 finds that human interpretations of activation-maximization images do not necessarily indicate unit selectivity. This supports a caution about relying on human interpretation, but the paper does not say that interpreters project prior beliefs onto models using unfamiliar concepts. Page numbers were not available in our copy. (LLM, conf=0.77)

  Experimental — accuracy not yet measured

On a theoretical level, many gradient-based attribution methods identify only a first-order approximation of the ideal attribution, which is sometimes a poor approximation.

“However, on a theoretical level, many gradientbased methods identify only a first-order approximation of the ideal attribution, which is sometimes a poor approximation (Watson, 2022).”

• cites Watson, 2022 sourcePartly supported

  Section 4.2.1 (“How local is ‘local’?”; page unavailable) says local linear approximations can become unreliable when the model is highly nonlinear. It discusses approximators such as LIME and SHAP, but does not establish that many gradient-based attribution methods provide only a first-order approximation of an ideal attribution. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Representations in language models might span multiple layers.

“Some work even suggests that representations in language models might span multiple layers (Yun et al., 2021;Lindsey et al., 2024).”

• cites Yun et al., 2021 sourceSupported
• cites Lindsey et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Single neurons are polysemantic.

“However, researchers discovered that single neurons are 'polysemantic' -they seem to respond to multiple kinds of features in both artificial (Wei et al., 2015;Nguyen et al., 2016c;Olah et al., 2017a) and biological networks (Churchland & Shenoy, 2007;Rigotti et al., 2013;Mante et al., 2013;Raposo et al., 2014).”

• cites Wei et al., 2015 sourceCould not verify

  The Introduction (page unavailable) says individual CNN neurons can correspond to texture patterns or object detectors, while intra-class knowledge is captured collectively by groups of neurons. Neither that passage nor the passages on neural pathways and class variation say that a single neuron responds to multiple kinds of features. I searched for single-neuron responses to multiple features and for polysemanticity. (LLM, conf=0.72)

  Experimental — accuracy not yet measured
• cites Nguyen et al., 2016cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Olah et al., 2017a sourceSupported

From an information-theoretic point of view, an arbitrarily powerful probe measures the mutual information between a hidden representation and a concept.

“Indeed, from an information-theoretic point of view, an arbitrarily powerful probe measures the mutual information between a hidden representation and a concept (Hewitt & Liang, 2019;Pimentel et al., 2020).”

• cites Hewitt & Liang, 2019 sourcePartly supported

  The Introduction says that a sufficiently expressive probe with enough training data can learn any task from a lossless representation. That supports the point about probe power, but our copy does not say that a probe measures mutual information between a representation and a concept. (LLM, conf=0.80)

  Experimental — accuracy not yet measured
• cites Pimentel et al., 2020 sourceNot checked yet

Probes can be successfully trained on hidden activations that lack any causal connection to the output, only localizing correlated hidden activation vectors.

“Probes can be successfully trained on hidden activations that lack any causal connection to the output, only localizing correlated hidden activation vectors.”

• cites Ravichander et al., 2021 sourcePartly supported

  Section 4.1 reports high probing accuracy for linguistic properties that the main task does not require. Section 6 says probing can reveal a correlated property in a representation but cannot determine whether the model relies on it. The paper does not establish that the hidden activations have no causal connection to the output or that probes localize particular activation vectors. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Geiger et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Elazar et al., 2021 sourcePartly supported

  The Introduction (p. 1) says successful probing shows that information is extractable from latent representations, not that the model uses it. Section 4 (pp. 4–5) reports properties that probes detect but that the model does not appear to use for word prediction. The claim overstates this distinction: the paper questions the causal role of particular probed properties, not whether the hidden activations themselves have any causal connection to the output. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

In the language of causality, unembedding a residual stream vector measures the direct effect of that vector on the output.

“In the language of causality (Pearl, 2009), unembedding a residual stream vector measures the direct effect of that vector on the output (McGrath et al., 2023).”

• cites Pearl, 2009Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites McGrath et al., 2023 sourcePartly supported

  Section 2.3.1 says unembedding corresponds to a direct effect when the RMSNorm factor is fixed to its forward-pass value. The paper’s derivation, “Unembedding-based impact with RMSnorm held constant approximates direct effect,” further says the measure generally approximates the direct effect and equals it for zero ablations. The claim omits those conditions and states the relationship for any residual-stream vector. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Interpretability research is often motivated by the engineering goal of understanding state-of-the-art models thoroughly enough to make assurances of their safety.

“For instance, interpretability research is often motivated by the engineering goal of understanding state-of-the-art models thoroughly enough to make assurances of their safety (Bereska & Gavves, 2024;Tegmark & Omohundro, 2023;Dalrymple et al., 2024).”

• cites Bereska & Gavves, 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Tegmark & Omohundro, 2023 sourceSupported
• cites Dalrymple et al., 2024 sourceSupported

Faster alternatives that work well in practice include gradient-based approximations to activation patching, such as attribution patching, AtP*, and integrated gradients.

“Faster alternatives that work well in practice (Marks et al., 2024;Templeton et al., 2024) include gradient-based approximations to activation patching, such as attribution patching (Nanda, 2023a;Syed et al., 2024), AtP* (Kramár et al., 2024), and integrated gradients (Sundararajan et al., 2017).”

• cites Marks et al., 2024 sourcePartly supported

  Section 2 (page not available) says attribution patching approximates activation patching for many components in parallel using two forward passes and one backward pass. It also describes an integrated-gradients approximation as more accurate but more expensive. The passages do not identify AtP* as a method used or evaluated by this paper, so the full list is not supported by Marks et al. alone. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Templeton et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Nanda, 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Recent research demonstrates that chains of thought are not entirely faithful to the model's underlying decision-making process.

“Recent research demonstrates that chains of thought are not entirely faithful to the model's underlying decision-making process (Agarwal et al., 2024;Atanasova et al., 2023;Turpin et al., 2023;Lanham et al., 2023;Ye & Durrett, 2022).”

• cites Agarwal et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Atanasova et al., 2023 sourcePartly supported

  The Summary and Outlook reports that all four tested setups of natural-language explanation models are prone to generate unfaithful explanations. This supports the broader point that explanations need not reflect a model’s decision-making process, but the paper does not specifically establish the claim for chains of thought. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Turpin et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Cammarata et al. (2020) validated their interpretation of a curve detector's function in a convolutional neural network by substituting its parts with simple handcrafted replacements.

“Cammarata et al. (2020)validated their interpretation of a curve detector's function in a convolutional neural network by substituting its parts with simple handcrafted replacements.”

• cites Cammarata et al., 2020 sourcePartly supported

  In its discussion of causality, *Curve Detectors* says feature visualization and attribution experiments support the interpretation that the neurons detect curves. It does not describe validating that interpretation by substituting the detector’s parts with handcrafted replacements; it says the algorithm was reimplemented by hand and points to the next article for that account. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

To improve the causal relevance of probe vectors, one approach is to use counterfactual data, which involves intervening on the concept of interest.

“To improve the causal relevance of probe vectors, one approach is to use counterfactual data, which involves intervening on the concept of interest (for instance, observing the resulting output if the dog in an image was changed to a cat) (Elazar et al., 2021;Mueller, 2024;Geiger et al., 2024b).”

• cites Elazar et al., 2021 sourcePartly supported

  The Introduction (pp. 1–2) proposes counterfactual intervention to assess whether a model uses a property, supporting the claim’s causal aim. But the intervention removes information from internal representations, not from counterfactual input data; the Introduction (p. 1) explicitly distinguishes it from changing inputs, so the dog-to-cat image example does not describe this paper’s approach. (LLM, conf=0.87)

  Experimental — accuracy not yet measured
• cites Mueller, 2024 sourcePartly supported

  Section 2.1 describes counterfactual inputs and interventions on model components as ways to assess causal effects. But Section 6 says it is not immediately clear how the paper's analysis applies to methods that derive projections for particular concepts in latent space. The paper therefore supports the general counterfactual-intervention idea, not the specific claim that counterfactual data improves the causal relevance of probe vectors. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Geiger et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Prior circuit discovery work has attempted to decompose models in either architectural bases or sparse autoencoder latents, which are imperfect ways to decompose neural networks for mechanistic interpretability.

“Perhaps most importantly, prior circuit discovery work has attempted to decompose models in either architectural bases (Wang et al., 2023;Conmy et al., 2023) or sparse autoencoder latents (Marks et al., 2024;Huben et al., 2024), which are imperfect ways to decompose neural networks for mechanistic interpretability (Section 2.1.2).”

• cites Wang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Conmy et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Marks et al., 2024 sourceSupported

Ensuring the safety of agentic systems with formal guarantees is complicated by the need to model a system's interactions in an arbitrarily complex environment that might not be formalizable.

“Ensuring the safety of agentic systems with formal guarantees is further complicated by the need to model a system's interactions in an arbitrarily complex environment that might not be formalizable (Seshia et al., 2022;Wongpiromsarn et al., 2023;Dalrymple et al., 2024).”

• cites Seshia et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wongpiromsarn et al., 2023 sourcePartly supported

  Section 1.3, “What Makes It Hard?” (page not specified), says uncertain, unstructured, changing environments make verification and synthesis more complex. The Introduction (page not specified) says formal guarantees depend on a well-defined model and notes the challenge of modeling open-world environments. Our copy does not establish that such environments may be impossible to formalize, or describe their complexity as arbitrary. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites Dalrymple et al., 2024 sourceSupported

Thus far, mechanistic interpretability methods that modify intermediate activations but not weights for unlearning have yet to yield competitive results.

“Thus far, mechanistic interpretability methods that modify intermediate activations (but not weights) for unlearning have yet to yield competitive results (Farrell et al., 2024).”

• cites Farrell et al., 2024 sourcePartly supported

  The Conclusion says the SAE method changes activations rather than weights and needs substantial improvement; it reports similar or greater unwanted side effects than weight-modifying RMU. But the results also report similar WMDP-bio unlearning and lower OpenWebText loss for the SAE method. The paper evaluates this particular method, not all mechanistic-interpretability methods, so the broader claim that none has yielded competitive results is not established. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Mechanistic interpretability decomposition methods enable steering models toward a narrower range of behaviors with fewer side effects.

“Mechanistic interpretability decomposition methods enable the steering of models toward a narrower range of behaviors with fewer side effects (Chalnev et al., 2024).”

• cites Chalnev et al., 2024 sourceSupported

Recent work suggests that existing finetuning methodologies primarily make shallow edits to existing model representations and circuitry.

“Recent work (Jain et al., 2024;Prakash et al., 2024;Lee et al., 2025) suggests that existing finetuning methodologies primarily make shallow edits to existing model representations and circuitry.”

• cites Jain et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Prakash et al., 2024 sourcePartly supported

  The Introduction (§1) says fine-tuning enhanced an existing entity-tracking mechanism rather than fundamentally shifting it, and found similar internal representations. The Discussion (§7) limits the study to one task and three fine-tuned models and says further work is needed to determine whether this invariance is typical. It therefore does not establish that existing fine-tuning methodologies generally make primarily shallow edits. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Lee et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Lin et al. (2024) used the refusal direction to red-team models by optimizing for inputs that minimize the projection of the residual stream onto this direction during the forward pass.

“Lin et al. (2024) used this to red-team models by optimizing for inputs that minimize the projection of the residual stream onto this direction during the forward pass.”

• cites Lin et al., 2024 sourcePartly supported

  Section 4 (Methodology; page unavailable) describes a jailbreak objective that maximizes a prompt representation’s projection along an acceptance direction in PCA-reduced space. The paper does not describe minimizing a residual-stream projection onto a refusal direction during the forward pass, so those specific mechanics are not supported. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

Program synthesis through mechanistic analysis offers an alternative approach by converting simple trained neural networks into more interpretable, controllable, and verifiable programs.

“Program synthesis through mechanistic analysis offers an alternative approach by converting simple trained neural networks into more interpretable, controllable, and verifiable programs (Michaud et al., 2024).”

• cites Michaud et al., 2024 sourcePartly supported

  The Introduction describes MIPS as distilling simple learned algorithms from neural networks into Python code. Section 3.3 says the resulting programs are more interpretable and verifiable, but our copy does not establish the additional claim that they are more controllable. No page numbers are provided in the indexed copy. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Interpretability research has thus far produced few tools that are used to make state-of-the-art systems safer in the real world.

“Nonetheless, interpretability research has thus far produced few tools that are used to make state-of-the-art systems safer in the real world (Rauker et al., 2023).”

• cites Rauker et al., 2023 sourceSupported

Human judgment might be required to determine which features are supposed to be relevant to the task.

“However, human judgment might be required to determine which features are 'supposed' to be relevant to the task (Marks et al., 2024;Kim et al., 2018a;Goyal et al., 2022).”

• cites Marks et al., 2024 sourceSupported
• cites Kim et al., 2018aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Goyal et al., 2022 sourceCould not verify

  In the label-association indicator section (page unavailable), the paper says that assessing whether certain predictions are correct requires manual visual inspection. That is different from deciding which features are supposed to be relevant to a task. Searches for task-relevant features and human selection found no passage addressing that decision. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Approaches inspired by mechanistic interpretability have been used to prove accuracy bounds on a single-layer transformer trained on a synthetic task, albeit with great difficulty.

“Approaches inspired by mechanistic interpretability have been used to prove accuracy bounds on a singlelayer transformer trained on a synthetic task, albeit with great difficulty (Gross et al., 2024).”

• cites Gross et al., 2024 sourceSupported

An understanding of model internals can help AI labs assess the risks from frontier models.

“An understanding of model internals can help AI labs assess the risks from frontier models (Chang et al., 2024;Shevlane et al., 2023;Casper et al., 2024) and thus better fulfill their obligations under the EU AI Act to”

• cites Chang et al., 2024 sourcePartly supported

  The Introduction (p. 2) says that evaluating LLMs helps identify their weaknesses and potential risks. Section 2.2 (p. 6) notes their poor interpretability, but the passages do not say that understanding model internals helps AI labs assess risks from frontier models. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites Shevlane et al., 2023 sourceNot checked — no full text of the source
• cites Casper et al., 2024 sourceSupported

It has been observed that new model capabilities can emerge as a function of scale.

“For instance, it has been observed that new model capabilities can emerge as a function of scale (Wei et al. (2022), though also see Schaeffer et al. (2023)).”

• cites Wei et al. (2022) sourceCould not verify

  Section 7 (Future directions; page unavailable) mentions power laws in neural language model training, but does not say that new capabilities emerge as models scale. Searches for emergent abilities and capabilities as a function of scale found no passage addressing that claim. (LLM, conf=0.74)

  Experimental — accuracy not yet measured
• cites Schaeffer et al. (2023)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The geometric arrangement of features in relation to each other seems to reflect semantic and functional structure.

“However, the geometric arrangement of features in relation to each other seems to reflect semantic and functional structure (Engels et al., 2024a;Gurnee & Tegmark, 2024;Park et al., 2024b;Bussman et al., 2024).”

• cites Engels et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Gurnee & Tegmark, 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Park et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The highly activating data set examples approach often yields plausible explanations for arbitrarily chosen directions in the activation space.

“A third issue is that this approach often yields plausible explanations for arbitrarily chosen directions in the activation space (Szegedy et al., 2014).”

• cites Szegedy et al., 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The latents identified by SDL depend on the data set used to train them.

“A complicating factor in using SDL to identify the learned mechanisms of neural networks is that the latents identified by depend on the data set used to train them.”

• cites Kissane et al., 2024cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Many of the issues with highly activating data set examples stem from the fact that they merely provide correlational explanations for the activation of a network component, rather than causal explanations.

“Many of the issues with highly activating data set examples stem from the fact that they merely provide correlational explanations for the activation of a network component, rather than causal explanations.”

Sparsity is not a good proxy for interpretability.

“The (related) problems of feature splitting (Bricken, 2023), feature absorption (Chanin et al., 2024) and composition (Till, 2024) suggest that with sufficient optimization pressure, sparsity as a proxy for interpretability breaks down (though note that it is debated whether feature splitting is actually a problem).”

• cites Bricken, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Chanin et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Till, 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It is possible to develop adversarial models that deliberately yield misleading feature visualizations.

“Furthermore, it is possible to develop adversarial models that deliberately yield misleading feature visualizations (Geirhos et al., 2025).”

• cites Geirhos et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Gaining insights about the network's mechanisms from SDL latents requires further post hoc analysis, which can be labor intensive, computationally expensive, or data set dependent.

“Gaining insights about the network's mechanisms from SDL latents requires further post hoc analysis, which can be labor intensive, computationally expensive, or data set dependent (Huben et al., 2024;Bricken, 2023;Riggs et al., 2023;Marks et al., 2024).”

• cites Huben et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Bricken, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Riggs et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Dimensionality reduction methods cannot identify more directions than there are activation dimensions.

“This is a key reason that dimensionality reduction methods are not considered state-of-the-art, because they cannot identify more directions than there are activation dimensions.”

Reconstruction errors are not purely random; much of the direction of the error and its norm can be linearly predicted from the initial activation vector.

“Engels et al. (2024b) found that these reconstruction errors are not purely random, as much of the direction of the error and its norm can be linearly predicted from the initial activation vector.”

• cites Engels et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Individual layers are too coarse-grained to robustly carve the network at its joints.

“This chimes with work that edits or intervenes on individual layers, which indicates that this level is too coarse-grained to robustly carve the network at its joints (Meng et al., 2022b;Wang et al., 2023).”

• cites Meng et al., 2022bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Dimensionality reduction methods are no longer predominant methods used for mechanistically decomposing language models.

“though these techniques are no longer predominant methods used for mechanistically decomposing language models (Friedman et al., 2024).”

• cites Friedman et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The causal approach to grounding mechanistic interpretability has not yet yielded canonical causal mediators that may form a basis for decomposition methods.

“However, this approach has not yet yielded canonical causal mediators (Mueller et al., 2024) that may form a basis for decomposition methods.”

• cites Mueller et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

When a sparse dictionary with 16 million latents was inserted into GPT-4, the language modeling loss was equivalent to a model with only 10% of GPT-4's pretraining compute.

“When a sparse dictionary with 16 million latents was inserted into GPT-4, the language modeling loss was equivalent to a model with only 10% of GPT-4's pretraining compute (Gao et al., 2024).”

• cites Gao et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Current SDL methods may systematically fail to capture certain structured aspects of model representations.

“This suggests that current SDL methods may systematically fail to capture certain structured aspects of model representations, but also implies potential solutions.”

SDL methods are expensive to apply to large models.

“SDL methods are expensive to apply to large models: SDL involves training a small neural network for every layer of the AI model that we want to interpret.”

SDL latents may not contain the concepts needed for downstream use cases.

“More often than not though, a sparse set of latents that encode some useful concept of interest do not exist.”

• cites Kantamneni et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Kissane et al., 2024cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Attention heads also exhibit polysemanticity.

“Interpreting individual attention heads does not fare better than interpreting individual neurons, as attention heads also exhibit polysemanticity (Janiak et al., 2023).”

• cites Janiak et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

SDL reconstruction errors are too high.

“SDL reconstruction errors are too high: Large errors in SDL reconstruction raise the question whether SDL methods can reconstruct the hidden representation sufficiently well such that the latents learned by SDL are adequately faithful to the models being interpreted.”

• cites Gao et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Makelov et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Current mechanistic interpretability work primarily focuses on understanding neural network activations, with little attention paid to how this structure in activations is computed via weights.

“This is an instance of a more broad problem with current mechanistic interpretability work; we primarily focus on understanding neural network activations, with little attention paid to how this structure in activations is computed via weights (Chughtai & Bushnaq, 2025).”

• cites Chughtai & Bushnaq, 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Highly activating data set examples have several substantial issues.

“Despite being widely used, this approach has several substantial issues.”

The weaker version of the linear representation hypothesis is supported by the successes of linear probes, activation steering, and sparse autoencoders.

“The weaker version of the linear representation hypothesis is supported by the successes of linear probes (Section 2.2.1), activation steering (Section 3.2.2),and the success of sparse autoencoders in finding seemingly interpretable latents (Paragraph 2.1.2c).”

Some gradient-based attribution methods identify attributions that are independent both of the model and of the data generating process.

“Adebayo et al. (2018) revealed more practical implications, demonstrating that some gradient-based methods identify attributions that are independent both of the model and of the data generating process.”

• cites Adebayo et al., 2018Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

As AI models become larger, scaling costs of SDL also increase.

“As AI models become larger, scaling costs of SDL also increase, although it remains unclear whether relative scaling costs are sub-or supra-linear.”

Depending on the data set from which examples were drawn, human annotators identified dramatically different meanings for given directions in the activation space of BERT.

“Depending on the data set from which the examples were drawn, human annotators identified dramatically different meanings for given directions in the activation space of BERT (Devlin, 2018).”

• cites Devlin, 2018Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Even when both humans and AI exhibit comparable levels of competence on a given task, they may use different heuristics.

“Even when both humans and AI exhibit comparable levels of competence on a given task, they may use different heuristics.”

• cites Geirhos et al., 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Brendel & Bethge, 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Nanda et al., 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Identifying relevant components through individual interventions is costly when there are many components.

“Identifying relevant components through individual interventions is costly when there are many components.”

Simple early circuits were found to be unfaithful.

“Simple early circuits were found to be unfaithful (Chan et al., 2022b;2023).”

• cites Chan et al., 2022bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [2023]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It is often hard to distinguish faithful explanations of neural network components from merely plausible ones.

“Unfortunately, it is often hard to distinguish faithful explanations of neural network components from merely plausible ones.”

• AssumptionRests on an assumption: It is often hard to distinguish faithful explanations of neural network components from merely plausible ones.

The logit lens cannot measure the indirect effect, the effects resulting from the influence that the embedding has on the hidden activations of subsequent layers.

“However, the logit lens cannot measure the indirect effect, the effects resulting from the influence that the embedding has on the hidden activations of subsequent layers.”

Predetermined explanations can be enforced at training time in conventional models using Interchange Intervention Training.

“Alternatively, predetermined explanations can be enforced at training time in conventional models using Interchange Intervention Training (Gupta et al., 2024;Geiger et al., 2022).”

• cites Gupta et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Geiger et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Existing faithfulness metrics are misleading when applied to several complex end-to-end circuits.

“Miller et al. (2024) show that existing measures of faithfulness depend on the causal intervention implementation used, and further demonstrate that such metrics are misleading when applied to several complex end-to-end circuits.”

• cites Miller et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The tasks studied so far have been deliberately selected to be simple to define and study mechanistically.

“The tasks studied so far have been deliberately selected to be simple to define and study mechanistically (Wang et al., 2023).”

• cites Wang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Concept-based probing can only identify concepts that we were already looking for, rather than reveal unexpected features in the network.

“This limitation implies that concept-based probing can only identify concepts that we were already looking for, rather than reveal unexpected features in the network.”

The selection of simple tasks gives a misleading impression of the level of difficulty involved in implementing circuit discovery for any arbitrary task that a network implements.

“This gives a misleading impression of the level of difficulty involved in implementing circuit discovery for any arbitrary task that a network implements.”

Conmy et al. (2023) validated a tool's ability to attribute model behaviors to internal components by running it on a simple model that implemented a known algorithm.

“For example, Conmy et al. (2023) validated a tool's ability to attribute model behaviors to internal components by running it on a simple model that implemented a known algorithm.”

• cites Conmy et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Even Contrast-Consistent Search requires the construction of data sets with clear positive and negative cases.

“Despite being more unsupervised than standard concept-based probing, even CCS requires the construction of data sets with clear positive and negative cases.”

• cites Burns et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Chain-of-thought does not use model internals in its explanations, at least not directly.

“While this could be considered an 'interpretability technique' in the sense that it aims to explain model decisions, it does not use model internals in those explanations, at least not directly.”

Networks can be trained to use particular causal structures instead of specific concepts.

“Instead of specific concepts, networks can also be trained to use particular causal structures (Geiger et al., 2022).”

• cites Geiger et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Unless interpretability methods demonstrate that they are competitive with alternative approaches to achieve engineering goals, the act of demonstrating their usefulness may lead to a bias toward developing methods that only perform well in best-case scenarios and on simple tasks, rather than those that can handle worst-case scenarios and practical challenges.

“Unless interpretability methods demonstrate that they are competitive with alternative approaches to achieve engineering goals, then the act of demonstrating their usefulness may lead to a bias toward developing methods that only perform well in best-case scenarios and on simple tasks, rather than those that can handle worst-case scenarios and practical challenges.”

Probes are tools for a correlational analysis, measuring if hidden activations serve as signals for a given concept.

“Probes are tools for a correlational analysis, measuring if hidden activations serve as signals for a given concept.”

ACDC automates part of the circuit discovery pipeline to identify computational subgraphs involved in particular tasks.

“An approach called Automated Circuit DisCovery' (ACDC) automates part of the pipeline discussed in Section 2.3 to identify computational subgraphs involved in particular tasks (Conmy et al., 2023).”

• cites Conmy et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Causal intervention methods can generate supervision signal to identify subspaces of interest.

“Causal intervention methods can also generate supervision signal to identify subspaces of interest.”

• cites Geiger et al., 2024cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Guerner et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wu et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The practice of conducting evaluations using non-cherry-picked tasks remains relatively uncommon.

“However, the practice of conducting evaluations using non-cherry-picked tasks remains relatively uncommon.”

• AssumptionRests on an assumption: The practice of conducting evaluations using non-cherry-picked tasks remains relatively uncommon.

Such models can be created by compiling simple programs into weights of models that exactly implement the known program.

“Such models can be created by compiling simple programs into weights of models that exactly implement the known program (Lindner et al., 2023;Weiss et al., 2021;Thurnherr & Scheurer, 2024;Gupta et al., 2024).”

• cites Lindner et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Weiss et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Thurnherr & Scheurer, 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Existing measures of faithfulness depend on the causal intervention implementation used.

“Miller et al. (2024) show that existing measures of faithfulness depend on the causal intervention implementation used, and further demonstrate that such metrics are misleading when applied to several complex end-to-end circuits.”

• cites Miller et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Other methods, such as causal interventions, are necessary to measure the indirect effect.

“Other methods, such as causal interventions (see below), are necessary to measure the indirect effect.”

If the weights of a toy neural network were handcrafted by humans, it is possible to obtain a ground truth explanation for how it works.

“If the weights of a toy neural network were handcrafted by humans, it is possible to obtain a ground truth explanation for how it works.”

Attempts to use techniques in the current interpretability toolkit in such evaluations have not proven to be consistently useful.

“Although attempts have been made to use techniques in the current interpretability toolkit in such evaluations, they have not proven to be consistently useful (Adebayo et al., 2020;Denain & Steinhardt, 2023;Casper et al., 2022b;Hase et al., 2023;Durmus et al., 2024).”

• cites Adebayo et al., 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Denain & Steinhardt, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Casper et al., 2022bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Attempts to study arbitrary circuits have proceeded less successfully.

“Indeed, attempts to study arbitrary circuits have proceeded less successfully (Nanda et al., 2023b).”

• cites Nanda et al., 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Thus far, researchers have mostly used model organisms that possess only some of the criteria for an ideal model organism, such as a transformer that can perform modular addition or GPT-2.

“Thus far, researchers have mostly used model organisms that possess only some of these criteria, such as a transformer than can perform modular addition (Nanda et al., 2023a) or GPT-2 (Radford et al., 2018).”

• cites Nanda et al., 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Radford et al., 2018Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Benchmarks can be established using models with known ground truth explanations.

“Benchmarks can be established using models with known ground truth explanations.”

Describing the functional role of model components is amenable to automation.

“A task that is amenable to automation is 'describing the functional role of model components' (Section 2.1.3).”

By using natural language explanations of a given network component's function, it is possible to predict the component's activation levels on different inputs.

“By using natural language explanations of a given network component's function, it is possible to predict the component's activation levels on different inputs.”

• cites Hernandez et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Bills et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Shaham et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It is feasible to leverage concepts in the network training process itself, as in concept bottleneck models.

“it is also feasible to leverage the concepts in the network training process itself, such as in the case of concept bottleneck models (Koh et al., 2020).”

• cites Koh et al., 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Certain choices made while studying model organisms risk steering the field in suboptimal directions.

“Furthermore, certain choices made while studying model organisms risk steering the field in suboptimal directions.”

Training networks with concepts makes it more likely that the components to which concepts are assigned are causally relevant.

“This is beneficial as it is more likely that the components to which concepts are assigned are causally relevant.”

Probing approaches face a considerable risk of not only discovering correlated features, but also spurious correlations due to the high dimensionality of hidden activations.

“However, probing approaches face a considerable risk of not only discovering correlated features, but also spurious correlations due to the high dimensionality of hidden activations.”

Fully automating current mechanistic interpretability pipelines would not yield satisfactory explanations of model behavior.

“While significant progress has been made toward automating steps of mechanistic interpretability pipelines, fully automating current pipelines would not yield satisfactory explanations of model behaviorfoot_5 .”

Finetuning can mask capabilities present in a base model in a way that can easily be reversed via simple changes to the model.

“Relatedly, (Jain et al., 2024) and Lee et al. (2025) show that finetuning can mask capabilities present in a base model in a way that can easily be reversed via simple changes to the model.”

• cites Jain et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lee et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Existing finetuning methodologies making shallow edits suggests that harmlessness training may be cheaply undone.

“Importantly, this suggests that harmlessness training (which trains models to refuse to answer harmful requests) may be cheaply undone.”

• Also stated in: § 3.2.2 Using mechanistic interpretability for better control of AI system behavior · boosted

  “Empirical evidence supports this claim, both with further finetuning (Gade et al., 2024;Lermen et al., 2024), as well as with causal interventions on the forward pass (Arditi et al., 2024).”

• cites Gade et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lermen et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Arditi et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Selective transparency can be used to actively mislead.

“Unfortunately, selective transparency can be used to actively mislead (Ananny & Crawford, 2018).”

• cites Ananny & Crawford, 2018 sourceNot checked — no full text of the source

The level of understanding of AI necessary to enable formal verification of large, general AI systems for nontrivial properties is well beyond the current capabilities of mechanistic interpretability.

“The level of understanding of AI necessary to enable formal verification of large, general AI systems for nontrivial properties is well beyond the current capabilities of mechanistic interpretability.”

Even shallow, correlation-based descriptions could signal potentially concerning cognition.

“Even shallow, correlation-based descriptions could signal potentially concerning cognition.”

Formal verification of AI systems remains an unresolved issue at present.

“Formal verification of AI systems remains an unresolved issue at present.”

Approaches incorporating SDL into safety cases have difficulty validating whether learned features capture all concerning patterns of reasoning reliably.

“While such approaches show promise, they have difficulty in validating whether learned features capture all concerning patterns of reasoning reliably.”

In synthetic data settings, the emergence of new capabilities coincides with abrupt changes in the trajectory of model parameters.

“Evidence suggests that new capabilities may be learned in a somewhat discrete Michaud et al. (2023) or stagewise (Wang et al., 2024b) fashion, and that in synthetic data settings, the emergence of new capabilities coincides with abrupt changes in the trajectory of model parameters (Park et al., 2024a).”

• cites Park et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The recent progress in mechanistic interpretability has been used to lobby against specific AI regulation by falsely claiming that recent advancements have resolved the black-box issue.

“Meanwhile, the recent definite -but ultimately modest -progress in mechanistic interpretability has been used to lobby against specific AI regulation by falsely claiming that,”

• cites Andreessen Horowitz, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The vast majority of mechanistic interpretability research to date has focused on just three model families: CNN-based image models, BERT-based text models, and GPT-based text models.

“The vast majority of mechanistic interpretability research to date has focused on just three model families: CNN-based image models (e.g.Erhan et al. (2009); Nguyen et al. (2016c); Olah et al. (2020b)), BERT-based text models (e.g.Devlin (2018); Rogers et al. (2020)) and GPT-based text models (e.g.Elhage et al. (2021); Wang et al. (2023); Nanda et al. 2023a).”

• cites Erhan et al. (2009)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Devlin (2018)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Elhage et al. (2021)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Current frontier AI governance efforts rarely specify concrete ways in which a mechanistic understanding of AI models might be used to help implementation.

“Current frontier AI governance efforts rarely specify concrete ways in which a mechanistic understanding of AI models might be used to help implementation.”

• cites OpenAI, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Despite current shortcomings in decomposition and description, white-box evaluations are likely feasible today.

“Despite current shortcomings in decomposition and description, white-box evaluations are likely feasible today.”

A correlation-based method that flags both features that recognize deceptive behavior and mechanisms that cause deceptive behavior can facilitate catching the latter.

“However, a correlation-based method that flags both can facilitate catching the latter.”

Activation steering is a control method derived from mechanistic interpretability insights.

“One new control method derived from mechanistic interpretability insights is activation steering (a.k.a.activation addition).”

• cites Li et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Turner et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Zou et al., 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The success of activation steering can be thought of as evidence for the Linear Representation Hypothesis.

“This technique results from the hypothesis, and its success can be thought of as evidence for the hypothesis.”

Moderate success can be achieved in steering using basic decomposition and description methods.

“Moderate success can be achieved in steering using basic decomposition and description methods.”

Optimizing for inputs that minimize the projection onto the refusal direction may be more efficient than optimizing over the whole model.

“This approach may be more efficient than optimizing over the whole model, as previous methods did (Zou et al., 2023b).”

• cites Zou et al., 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Evidence suggests that new capabilities may be learned in a somewhat discrete or stagewise fashion.

“Evidence suggests that new capabilities may be learned in a somewhat discrete Michaud et al. (2023) or stagewise (Wang et al., 2024b) fashion, and that in synthetic data settings, the emergence of new capabilities coincides with abrupt changes in the trajectory of model parameters (Park et al., 2024a).”

• cites Michaud et al. (2023)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wang et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Current methods allow for versions of microscope AI, depending on the kind of insights that we want to learn.

“Current methods allow for versions of microscope AI, depending on the kind of insights that we want to learn.”

• cites Schut et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ludwig & Mullainathan, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Tong et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

By understanding the mechanisms of jailbreaking, we can anticipate the means through which a user might bypass existing safeguards.

“By understanding the mechanisms of jailbreaking, we can anticipate the means through which a user might bypass existing safeguards (Lee et al., 2025;Arditi et al., 2024).”

• cites Lee et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Arditi et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Localizing knowledge and capabilities within models may improve the sample efficiency of finetuning by selectively modifying only relevant parameters.

“Separately, localizing knowledge and capabilities within models may improve the sample efficiency of finetuning, by selectively modifying only relevant parameters (as in, e.g.Wu et al. (2024b)).”

• cites Wu et al. (2024b)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.