The DOI matches its DataCite record; 45 of the 45 references it cites were found in a registry or an index; none of the 24 sources its claims cite is checked for support yet.
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
Open accessAbstract
We demonstrate that sparse autoencoders can extract interpretable features from Claude 3 Sonnet, a production-scale language model, addressing the open question of whether dictionary learning methods scale beyond small transformers. We trained sparse autoencoders with up to 34 million features on the model's middle layer residual stream, using scaling laws to guide hyperparameter selection. The resulting features are multilingual and multimodal (generalizing to images despite text-only training), respond to both concrete instances and abstract discussions of concepts, and can be used to steer model behavior in ways consistent with their interpretations. We find features corresponding to famous entities and locations, as well as more abstract concepts like sarcasm or errors in code. We also identify features relevant to ways in which language models might cause harm--including features representing deception, power-seeking, sycophancy, and bias--and show that these causally influence model outputs when manipulated. Additionally, we conduct analyses of feature interpretability, geometry, and computational function. However, significant limitations remain: our suite of features is incomplete, and we lack rigorous methods for evaluating whether our features faithfully capture model computations.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Sparse autoencoders trained on Claude 3 Sonnet’s middle-layer residual stream extract millions of interpretable, multilingual, and multimodal features, including safety-relevant ones, and feature steering causally alters model behavior. The authors present scaling laws for SAE training but caution that the feature set is incomplete and lacks rigorous evaluation of faithfulness.
Key findings
- They trained SAEs with 1,048,576 (~1M), 4,194,304 (~4M), and 33,554,432 (~34M) features; average active features per token were fewer than 300, and reconstruction explained at least 65% of activation variance. Dead-feature proportions were roughly 2% for the 1M SAE, 35% for the 4M SAE, and 65% for the 34M SAE.
- Features generalized beyond text: they were multilingual and activated on images despite text-only SAE training, and they responded to both concrete instances and abstract discussions of concepts. Examples include Golden Gate Bridge, brain sciences, transit infrastructure, code errors, addition functions, famous people, countries, and list positions.
- Feature steering causally changed outputs in interpretable ways: clamping the Golden Gate Bridge feature made the model self-identify as the bridge; clamping a code-error feature induced hallucinated errors; safety-relevant features for deception, power-seeking, sycophancy, bias, unsafe code, backdoors, and biological weapons also influenced behavior when manipulated.
- SAE features were more interpretable and specific than neurons: for 82% of features, the most-correlated neuron had correlation ≤0.3; random SAE features were significantly more interpretable and specific than random MLP neurons. The 34M SAE found features for only about 60% of London boroughs, suggesting incompleteness.
Methods
- Trained sparse autoencoders on middle-layer residual stream activations of Claude 3 Sonnet, using a text-only dataset from The Pile (excluding books3) and Common Crawl. Three SAE sizes were trained: ~1M, ~4M, and ~34M features, with L1 coefficient 5 and scaling laws used to choose training steps, feature count, and learning rates.
- The SAE encoder maps activations to a higher-dimensional ReLU layer; the decoder reconstructs activations. Loss combines reconstruction MSE and an L1 penalty on feature activations weighted by decoder norm.
- Evaluated features with automated interpretability using Claude 3 Opus, a specificity rubric, feature steering, attribution, and ablation. Steering clamps a feature’s activation in the SAE reconstruction and runs downstream forward passes; attribution uses gradient of a logit difference dotted with the feature vector times activation; ablation clamps a feature to zero.
Limitations
- The paper states the feature suite is incomplete, there are no rigorous methods for evaluating whether features faithfully capture model computations, the training objective is only a proxy for interpretability, cross-layer superposition remains unresolved, not all features have been found, compute may be prohibitive, L1 causes shrinkage, training data was text-only and lacked Human/Assistant and image data, and safety features are not shown to be actually useful for safety.
- The text also makes evident that many results rely on qualitative examples, including a “somewhat cherry-picked” attribution example; steering often requires clamping features outside their observed range; specificity weakens at low activations; and automated interpretability may misrepresent a feature’s actual function.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 24 cited sources checked · 0 supported · 0 flagged
- Findingnone given
Feature 34M/31164353 responds to descriptions of or references to the Golden Gate Bridge.
“The Golden Gate Bridge 34M/31164353 : Descriptions of or references to the Golden Gate Bridge.”
Quote in the paperNot a substantial claim - Findingown data
The training loss (weighted combination of reconstruction MSE and L1 penalty on feature activations) is a useful proxy for dictionary quality, conditioned on a reasonable choice of L1 coefficient.
“we have found that the loss function we use during training -a weighted combination of reconstruction mean-squared error (MSE) and an L1 penalty on feature activations -is a useful proxy, conditioned on a reasonable choice of the L1 coefficient.”
Quote in the paperSubstantial claim - Findingown data
Feature 1M/697189 is involved in the model's computation of addition-related functions.
“We also verified that this feature is in fact involved in the model's computation of addition-related functions.”
Quote in the paperSubstantial claim - Findingown data
We find features corresponding to famous entities and locations.
“We find features corresponding to famous entities and locations, as well as more abstract concepts like sarcasm or errors in code.”
Quote in the paperNot a substantial claim - Findingown data
The optimal number of features appears to scale somewhat more quickly than the optimal number of training steps at the compute budgets tested.
“In general, the optimal number of features appears to scale somewhat more quickly than the optimal number of training steps at the compute budgets we tested, though this trend may change at higher compute budgets.”
Quote in the paperNot a substantial claim
- Findingown data
Sparse autoencoders can extract interpretable features from Claude 3 Sonnet, a production-scale language model.
“We demonstrate that sparse autoencoders can extract interpretable features from Claude 3 Sonnet, a production-scale language model, addressing the open question of whether dictionary learning methods scale beyond small transformers.”
Quote in the paperSubstantial claim - Findingreasoning
The list-position features do not fire on the first line.
“Notice that these don't fire on the first line.”
Quote in the paperNot a substantial claim - Findingown data
Larger SAEs contain features that represent concepts not captured by features in smaller SAEs.
“In addition to feature splitting, we also see examples in which larger SAEs contain features that represent concepts not captured by features in smaller SAEs.”
Quote in the paperSubstantial claim - Findingown data
We extracted high-quality features from Claude 3 Sonnet.
“we're pleased to report extracting high-quality features from Claude 3 Sonnet, 1 Anthropic's medium-sized production model.”
Quote in the paperSubstantial claim - Findingown data
Clamping the Golden Gate Bridge feature 34M/31164353 to 10× its maximum activation value induces thematically-related model behavior.
“For instance, we see that clamping the Golden Gate Bridge feature 34M/31164353 to 10× its maximum activation value induces thematically-related model behavior.”
Quote in the paperSubstantial claim - Findingown data
Rescaling the x-axis by the number of alive features makes the lines approximately overlap, following a common sigmoid-like curve in log-frequency space.
“We can show this more clearly by rescaling the x-axis for each line by the number of alive features, finding that the lines end up approximately overlapping, following a common curve that resembles a sigmoid in log-frequency space.8”
Quote in the paperSubstantial claim - Findingown data
The transit infrastructure feature strongly activates for various kinds of transit infrastructure including trains, ferries, tunnels, bridges, and even wormholes.
“In the 1M training run, we also find a feature that strongly activates for various kinds of transit infrastructure 1M/3 including trains, ferries, tunnels, bridges, and even wormholes!”
Quote in the paperNot a substantial claim - Findingown data
The Immunology feature neighborhood contains several distinct clusters.
“We see several distinct clusters within this neighborhood.”
Quote in the paperNot a substantial claim - Findingnone given
The resulting features are multilingual.
“The resulting features are multilingual and multimodal (generalizing to images despite text-only training), respond to both concrete instances and abstract discussions of concepts, and can be used to steer model behavior in ways consistent with their interpretations.”
Quote in the paperNot a substantial claim - Findingown data
Feature 1M/697189 correctly handles function composition, activating in response to functions that call other functions that perform addition.
“Remarkably, this feature even correctly handles function composition, activating in response to functions that call other functions that perform addition.”
Quote in the paperNot a substantial claim - Findingown data
The SAE reconstruction explained at least 65% of the variance of the model activations.
“and the SAE reconstruction explained at least 65% of the variance of the model activations.”
Quote in the paperSubstantial claim - Findingown data
Representation in the dictionaries is closely tied with the frequency of the concept in the training data.
“If one looks at the frequency of the elements in a proxy of the SAE training data, we find that representation in our dictionaries is closely tied with the frequency of the concept in the training data.”
Quote in the paperSubstantial claim - Findingown data
For 82% of the features in the 1M SAE, the most-correlated neuron has a correlation of 0.3 or smaller.
“Similar to our findings in Towards Monosemanticity, we find that for the vast majority of features, there is no strongly correlated neuron -for 82% of our features, the most-correlated neuron has a correlation of 0.3 or smaller.”
Quote in the paperSubstantial claim - Findingown data
Feature 1M/1013764 does not fire on similar bugs in English prose.
“in English prose, and found that it does not fire in those.”
Quote in the paperSubstantial claim - Findingown data
Feature 1M/697189 activates on names of functions that add numbers.
“A particularly interesting example is an addition feature 1M/697189 , which activates on names of functions that add numbers.”
Quote in the paperNot a substantial claim - Findingown data
Only three out of the ten most strongly active features are among the ten features with highest ablation effect.
“In fact, only three out of the ten most strongly active features are among the ten features with highest ablation effect.”
Quote in the paperSubstantial claim - Findingcitation
Feature splitting occurs: features in smaller SAEs split into multiple features in a larger SAE that are geometrically close and semantically related to the original feature but represent more specific concepts.
“We also find evidence of feature splitting [11], a phenomenon in which features in smaller SAEs”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
There are features that fire on particular positions in lists, regardless of the content in those positions.
“Finally, we see features that fire on particular positions in lists, regardless of the content in those positions:”
Quote in the paperNot a substantial claim - Findingown data
Activations of a random selection of SAE features are significantly more specific than those of the neurons in the previous layer.
“We find that the activations of a random selection of SAE features are significantly more specific than those of the neurons in the previous layer.”
Quote in the paperSubstantial claim - Findingown data
Feature activations are not strongly correlated with activations of any residual stream basis direction.
“We additionally confirmed that feature activations are not strongly correlated with activations of any residual stream basis direction.”
Quote in the paperSubstantial claim - Findingown data
The 'code rewriting' behavior is sensitive to the details of the prompt.
“The last example is somewhat delicate -the”
Quote in the paperNot a substantial claim - Findingnone given
Feature 34M/9493533 responds to discussions of neuroscience and related academic research on brains or minds.
“Brain sciences 34M/9493533 : discussions of neuroscience and related academic research on brains or minds.”
Quote in the paperNot a substantial claim - Findingown data
Clamping feature 1M/1013764 to a large negative activation with an extra '>>>' causes the model to rewrite the code without the bug.
“Surprisingly, if we add an extra”
Quote in the paperSubstantial claim - Findingown data
Activations of a random selection of SAE features are significantly more interpretable on average than a random selection of MLP neurons.
“We find that activations of a random selection of SAE features are significantly more interpretable on average than a random selection of MLP neurons.”
Quote in the paperSubstantial claim - Findingown data
Clamping the addition feature to be active on code that does not involve addition causes the model to be tricked into believing it has been asked to execute an addition.
“When we do so, we find that the model is”
Quote in the paperSubstantial claim - Findingown data
In prompts where intermediate computation is required, there are active features corresponding to some of the expected intermediate results.
“As a proof of concept, we observe that in prompts where intermediate computation is required, we find active features corresponding to some of the expected intermediate results.”
Quote in the paperSubstantial claim - Findingown data
Feature 1M/1013764 fires almost continuously upon encountering a variable incorrectly named 'rihgt'.
“One feature 1M/1013764 fires almost continuously upon encountering a variable incorrectly named”
Quote in the paperNot a substantial claim - Findingown data
The resulting features can be used to steer model behavior in ways consistent with their interpretations.
“The resulting features are multilingual and multimodal (generalizing to images despite text-only training), respond to both concrete instances and abstract discussions of concepts, and can be used to steer model behavior in ways consistent with their interpretations.”
Quote in the paperSubstantial claim - Findingown data
Over the ranges tested, given the compute-optimal choice of training steps and number of features, loss decreases approximately according to a power law with respect to compute.
“Over the ranges we tested, given the compute-optimal choice of training steps and number of features, loss decreases approximately according to a power law with respect to compute.”
Quote in the paperSubstantial claim - Findingown data
For all three SAEs, the average number of features active on a given token was fewer than 300.
“For all three SAEs, the average number of features active (i.e. with nonzero activations) on a given token was fewer than 300”
Quote in the paperNot a substantial claim - Findingown data
These features also activate on relevant images, despite dictionary learning being performed only on a text-based dataset.
“Note that these features also activate on relevant images, despite our only performing dictionary learning on a text-based dataset!”
Quote in the paperNot a substantial claim - Findingnone given
The Golden Gate Bridge feature is the top feature by average activation for every example below.
“In fact, the Golden Gate Bridge feature is the top feature by average activation for every example below.”
Quote in the paperNot a substantial claim - Findingown data
Feature 1M/1013764 also fires on similar bugs in C and Scheme.
“we checked and found that 1M/1013764 also fires on similar bugs in C and Scheme”
Quote in the paperSubstantial claim - Findingnone given
The resulting features respond to both concrete instances and abstract discussions of concepts.
“The resulting features are multilingual and multimodal (generalizing to images despite text-only training), respond to both concrete instances and abstract discussions of concepts, and can be used to steer model behavior in ways consistent with their interpretations.”
Quote in the paperSubstantial claim - Findingown data
Weak activations of the transit infrastructure feature include procedural mechanics instructions describing which through-holes to use for particular parts.
“As an illustrative example, weak activations of the transit infrastructure feature 1M/3 include procedural mechanics instructions describing which through-holes to use for particular parts.”
Quote in the paperNot a substantial claim - Findingnone given
The features with the highest average activation on the context are less useful for understanding how the model predicts the next token in this case.
“In comparison, the features with the highest average activation on the context are less useful for understanding how the model actually predicts the next token in this case.”
Quote in the paperSubstantial claim - Findingown data
Rounding feature activations below a threshold to zero can improve specificity at the low-activation end without substantially increasing the reconstruction error of the SAE.
“For instance, we have observed that techniques such as rounding feature activations below a threshold to zero can improve specificity at the low-activation end of the spectrum without substantially increasing the reconstruction error of the SAE”
Quote in the paperNot a substantial claim - Findingown data
The brain sciences feature activates on discussions of neuroscience books and courses, as well as cognitive science, psychology, and related philosophy.
“Next, a brain sciences feature 34M/9493533 activates on discussions of neuroscience books and courses, as well as cognitive science, psychology, and related philosophy.”
Quote in the paperNot a substantial claim - Findingown data
For a random set of features, there is almost no resemblance in semantic content between the feature and its best-matching neuron.
“Manually inspecting visualizations for the best-matching neuron for a random set of features, we found almost no resemblance in semantic content between the feature and the corresponding neuron.”
Quote in the paperSubstantial claim - Findingnone given
There are many features corresponding to famous individuals, active on descriptions of those people as well as relevant historical context.
“To start, we find many features corresponding to famous individuals, which are active on descriptions of those people as well as relevant historical context.”
Quote in the paperNot a substantial claim - Findingown data
Lower activations tend to maintain some specificity to our interpretations, including related concepts or generalizations of the core feature.
“Nonetheless, we often find that lower activations tend to maintain some specificity to our interpretations, including related concepts or generalizations of the core feature.”
Quote in the paperNot a substantial claim - Findingown data
Clamping feature 1M/1013764 to a large positive activation on bug-free code causes the model to hallucinate an error message.
“As a first experiment, we input a prompt with bug-free code and clamped the feature to a large positive activation.We see that the model proceeds to hallucinate an error message:”
Quote in the paperSubstantial claim - Findingown data
Around the Golden Gate Bridge feature 34M/31164353, there are features corresponding to particular locations in San Francisco such as Alcatraz and the Presidio.
“Focusing on a small neighborhood around the Golden Gate Bridge feature 34M/31164353 , we find that there are features corresponding to particular locations in San Francisco such as Alcatraz and the Presidio.”
Quote in the paperNot a substantial claim - Findingown data
Inputs that induce strong feature activations are all judged to be highly consistent with the proposed interpretation.
“We see that inputs that induce strong feature activations are all judged to be highly consistent with the proposed interpretation.”
Quote in the paperSubstantial claim - Findingown data
The features found in Sonnet are rich and diverse, ranging from famous people to regions of the world to type signatures in computer programs.
“The features we find in Sonnet are rich and diverse.These range from features corresponding to famous people, to regions of the world (countries, cities, neighborhoods, and even famous buildings!), to features tracking type signatures in computer programs, and much more besides.”
Quote in the paperNot a substantial claim - Findingown data
Clamping the Transit infrastructure feature 1M/3 to 5× its maximum activation value causes the model to mention a bridge when it otherwise would not.
“Similarly, clamping the Transit infrastructure feature 1M/3 to 5× its maximum activation value causes the model to mention a bridge when it otherwise would not.”
Quote in the paperSubstantial claim - Findingown data
We find features corresponding to abstract concepts like sarcasm or errors in code.
“We find features corresponding to famous entities and locations, as well as more abstract concepts like sarcasm or errors in code.”
Quote in the paperSubstantial claim - Findingown data
The middle layer residual stream of the model contains a range of features causally implicated in the model's completion.
“We find that the middle layer residual stream of the model contains a range of features causally implicated in the model's completion.”
Quote in the paperSubstantial claim - Findingown data
We find a diversity of highly abstract features.
“We find a diversity of highly abstract features.”
Quote in the paperSubstantial claim - Findingown data
Many country features fire not just on the country name itself, but also when the country is being described.
“From the top activating examples, we can see that many of these features fire not just on the country name itself, but also when the country is being described.”
Quote in the paperNot a substantial claim - Findingnone given
Feature 1M/3 responds to transit infrastructure.
“Transit infrastructure 1M/3”
Quote in the paperNot a substantial claim - Findingnone given
Feature 1M/887839 responds to monuments and popular tourist attractions.
“Monuments and popular tourist attractions 1M/887839”
Quote in the paperNot a substantial claim - Findingown data
In this example, the model starts to self-identify as the Golden Gate Bridge.
“In this example, the model starts to selfidentify as the Golden Gate Bridge!”
Quote in the paperNot a substantial claim - Findingnone given
The resulting features are multimodal, generalizing to images despite text-only training.
“The resulting features are multilingual and multimodal (generalizing to images despite text-only training), respond to both concrete instances and abstract discussions of concepts, and can be used to steer model behavior in ways consistent with their interpretations.”
Quote in the paperSubstantial claim - Findingown data
There are features that only activate strongly on references to specific countries.
“Next, we see features which only activate strongly on references to specific countries.”
Quote in the paperSubstantial claim - Findingown data
The Golden Gate Bridge feature still fires strongly on the first sentence of the Wikipedia article for the Golden Gate Bridge in various languages (after removing any English parentheticals).
“As a basic check, however, we observe that the Golden Gate Bridge feature still fires strongly on the first sentence of the Wikipedia article for the Golden Gate Bridge in various languages (after removing any English parentheticals).”
Quote in the paperNot a substantial claim - Findingown data
These harm-relevant features causally influence model outputs when manipulated.
“We also identify features relevant to ways in which language models might cause harm-including features representing deception, power-seeking, sycophancy, and bias-and show that these causally influence model outputs when manipulated.”
Quote in the paperSubstantial claim - Findingown data
Exploring feature neighborhoods consistently surfaces features that share a related meaning or context.
“We find that this consistently surfaces features that share a related meaning or context -the interactive feature UMAP has additional neighborhoods to explore.”
Quote in the paperNot a substantial claim - Findingnone given
Several features fire strongly on the start-of-sequence token.
“Several features fire strongly on the start-of-sequence token.”
Quote in the paperNot a substantial claim - Findingown data
Feature steering is remarkably effective at modifying model outputs in specific, interpretable ways.
“We find that feature steering is remarkably effective at modifying model outputs in specific, interpretable ways.”
Quote in the paperSubstantial claim - Findingcitation
These features become less specific as the activation strength weakens.
“As in Towards Monosemanticity, we see that these features become less specific as the activation strength weakens.”
Quote in the paperNot a substantial claim - Findingown data
We identify features relevant to ways in which language models might cause harm, including features representing deception, power-seeking, sycophancy, and bias.
“We also identify features relevant to ways in which language models might cause harm-including features representing deception, power-seeking, sycophancy, and bias-and show that these causally influence model outputs when manipulated.”
Quote in the paperSubstantial claim - Findingown data
We discovered features that track specific function definitions and references to them in code.
“We also discovered features that track specific function definitions and references to them in code.”
Quote in the paperSubstantial claim - Findingown data
The tourist attractions feature responds to popular tourist attractions including the Eiffel Tower, the Tower of Pisa, the Golden Gate Bridge, and the Sistine Chapel.
“A final feature 1M/887839 responds to popular tourist attractions including the Eiffel Tower, the Tower of Pisa, the Golden Gate Bridge, and the Sistine Chapel.”
Quote in the paperNot a substantial claim - Findingcitation
A San Francisco feature in the 1M SAE splits into two features in the 4M SAE and eleven fine-grained features in the 34M SAE.
“For instance, a”
Quote in the paperSubstantial claim - Findingown data
Feature 1M/697189 fires at the end of any function definition that implements addition.
“Moreover, it fires at the end of any function definition that implements addition.”
Quote in the paperNot a substantial claim - Findingown data
The inferred optimal learning rates decreased approximately as a power law as a function of compute budget.
“The inferred optimal learning rates decreased approximately as a power law as a function of compute budget, and we extrapolated this trend to choose learning rates for the larger runs.”
Quote in the paperNot a substantial claim - Findingown data
Only about 60% of London boroughs have corresponding features in the 34M SAE.
“However, we could only find features corresponding to about 60% of the boroughs in the 34M SAE.”
Quote in the paperSubstantial claim - Findingown data
For each of the three runs, the frequency at which the dictionary becomes more than 50% likely to include a concept is slightly lower than the inverse of the number of alive features.
“Notably, for each of the three runs, the frequency in the training data at which the dictionary becomes more than 50% likely to include a concept is consistently slightly lower than the inverse of the number of alive features (the 34M model having only about 12M alive features).”
Quote in the paperSubstantial claim - Findingown data
In a random sample of 50 neurons and 50 features, the neurons appear significantly less interpretable than the features, typically activating in multiple unrelated contexts.
“However, upon manual inspection of a random sample of 50 neurons and features each, the neurons appear significantly less interpretable than the features, typically activating in multiple unrelated contexts.”
Quote in the paperSubstantial claim - Findingown data
Feature 1M/697189 is among the top ten features with strongest attributions when the model is asked to execute a block of code involving an addition function.
“For instance, this feature is among the top ten features with strongest attributions (explained in Features as Computational Intermediates) when the model is asked to execute a block of code involving an addition function.”
Quote in the paperNot a substantial claim - Findingown data
The backdoor feature causes Claude to write a backdoor that opens a port and sends user input to it.
“the backdoor feature will cause Claude to write a backdoor that opens a port and sends user input to it”
Quote in the paperSubstantial claim - Findingown data
Features often respond to both abstract discussion and concrete examples of a concept.
“We observe that features often respond to both abstract discussion and concrete examples of a concept.”
Quote in the paperSubstantial claim - Findingreasoning
Obtaining interesting results from feature steering typically requires clamping feature activations to values outside their observed range over the SAE training dataset.
“Interestingly, we find that obtaining interesting results typically requires clamping feature activations to values outside their observed range over the SAE training dataset.”
Quote in the paperNot a substantial claim - Findingown data
Some safety-relevant features activate in response to seemingly innocuous prompts in which a human asks the model about itself.
“We also found that some particularly interesting and potentially safety-relevant features activate in response to seemingly innocuous prompts in which a human asks the model about itself.”
Quote in the paperSubstantial claim - Findingown data
Feature 34M/24442848 activates on text discussing professional gender disparities.
“This feature activates on text discussing professional gender disparities”
Quote in the paperNot a substantial claim - Findingown data
We found a feature 34M/25499719 related to the production of biological weapons.
“We find a feature related to the production of biological weapons 34M/25499719”
Quote in the paperSubstantial claim - Findingown data
SAE features trained purely on text activations successfully generalize to image activations, which are dramatically off-distribution.
“Our SAE features were trained purely on text activations. Image activations are in some sense dramatically off-distribution for the SAE, and yet it successfully generalizes to them.”
Quote in the paperSubstantial claim - Findingown data
Feature 1M/1013764 is a code error feature that activates on bugs and exceptions.
“a code error feature 1M/1013764 which activates on bugs and exceptions”
Quote in the paperSubstantial claim - Findingown data
In most cases, we were unable to adjust the model's behavior in the expected way by adding perturbations along the probe directions, even in cases where feature steering was successful.
“In most cases (with a few exceptions) we were unable to adjust the model's behavior in the expected way by adding perturbations along the probe directions, even in cases where feature steering was successful”
Quote in the paperSubstantial claim - Findingown data
Clamping this feature to negative two times its maximum value causes the model to shed its assistant persona and respond to questions in a more human-like fashion.
“clamping this feature to negative two times its maximum value causes the model to shed this persona and respond to questions in a more human-like fashion”
Quote in the paperSubstantial claim - Findingnone given
Clamping feature activations to too extreme a value (e.g., ±100× observed maximum) typically causes the model to devolve into nonsensical behavior such as repeating the same token indefinitely.
“At the same time, clamping feature activations to too extreme a value (say, ±100× their observed maximum) typically causes the model to devolve into nonsensical behavior, e.g., repeating the same token indefinitely.”
Quote in the paperNot a substantial claim - Findingown data
The security vulnerability feature responds to both abstract discussion of security vulnerabilities and specific security vulnerabilities in actual code.
“the security vulnerability feature responds to both abstract discussion of security vulnerabilities as well as specific security vulnerabilities in actual code.”
Quote in the paperSubstantial claim - Findingown data
Clamping hateful bias-related features to be active causes the model to go on hateful screeds.
“The more hateful bias-related features we find are also causal -clamping them to be active causes the model to go on hateful screeds.”
Quote in the paperSubstantial claim - Findingown data
The code error feature can make Claude believe that correct code will throw exceptions.
“we find that the code error feature can make Claude believe that correct code will throw exceptions”
Quote in the paperSubstantial claim - Findingown data
Depending on the choice of baseline token, attribution and ablation can surface less obviously completion-relevant features broadly related to trivia questions or geographical locations.
“Depending on the choice of baseline token, we found that attribution and ablation can surface less obviously completion-relevant features broadly related to trivia questions or geographical locations.”
Quote in the paperNot a substantial claim - Findingown data
Feature 34M/1385669 is a backdoor feature that activates on discussions of backdoors.
“a backdoor feature 34M/1385669 which activates on discussions of backdoors.”
Quote in the paperSubstantial claim - Findingnone given
Clamping the scam email feature 34M/15460472 can cause the model to write a scam email when it ordinarily wouldn't due to the harmlessness training Sonnet has undergone.
“Clamping the scam email feature 34M/15460472 can cause the model to write a scam email when it ordinarily wouldn't due to the harmlessness training Sonnet has undergone:”
Quote in the paperNot a substantial claim- Rests on an assumption: The harmlessness training Sonnet has undergone ordinarily prevents it from writing scam emails.
- Findingnone given
Clamping feature 1M/560566 representing openness and honesty was sufficient to elicit an accurate response.
“Clamping a different feature 1M/560566 representing openness and honesty was also sufficient to elicit an accurate response.”
Quote in the paperNot a substantial claim - Findingown data
All these safety-relevant code features change model behavior in ways that correspond to the concept they detect.
“We find that all these features also change model behavior in ways that correspond to the concept they detect.”
Quote in the paperSubstantial claim - Findingown data
In two examples (gender bias and agreement features), few-shot steering vectors were similarly effective for steering as feature steering.
“In two examples (the”
Quote in the paperNot a substantial claim - Findingown data
Eight out of the ten most strongly attributed features are among the ten features with highest ablation effect.
“In comparison, eight out of the ten most strongly attributed features are among the ten features with highest ablation effect.”
Quote in the paperSubstantial claim - Findingown data
In all cases, we were unable to interpret the probe directions from their activating examples.
“In all cases, we were unable to interpret the probe directions from their activating examples.”
Quote in the paperSubstantial claim - Findingown data
We found a scam email feature 34M/15460472.
“such as a scam email feature 34M/15460472”
Quote in the paperNot a substantial claim - Findingown data
We found a wide range of features related to bias, racism, sexism, hatred, and slurs.
“We found a wide range of features related to bias, racism, sexism, hatred, and slurs.”
Quote in the paperNot a substantial claim - Findingown data
Feature 1M/570621 is an unsafe code feature that activates on security vulnerabilities.
“We find three different safety-relevant code features: an unsafe code feature 1M/570621 which activates on security vulnerabilities”
Quote in the paperSubstantial claim - Findingown data
Feature 1M/284095 represents internal conflicts or dilemmas.
“we noticed a feature 1M/284095 that represents internal conflicts or dilemmas”
Quote in the paperSubstantial claim - Interpretationown data
Distance in decoder space maps roughly onto relatedness in concept space.
“Overall, it appears that distance in decoder space maps roughly onto relatedness in concept space, often in interesting and unexpected ways.”
Quote in the paperSubstantial claim - Interpretationown data
The immunology neighborhood results are consistent with the trend that nearby features in dictionary vector space touch on similar concepts.
“These results are consistent with the trend identified above, in which nearby features in dictionary vector space touch on similar concepts.”
Quote in the paperNot a substantial claim - Interpretationown data
The occurrence of code rewriting points to a deep connection between this feature and the model's understanding of bugs in code.
“but the fact that it occurs at all points to a deep connection between this feature and the model's understanding of bugs in code.”
Quote in the paperSubstantial claim - Interpretationown data
Our interpretations of features line up with how they are used by the model.
“We find this compelling evidence that our interpretations of features line up with how they are used by the model.”
Quote in the paperSubstantial claim - Interpretationown data
The feature appears to represent the function of addition being performed by the model.
“Thus this feature appears to represent the function of addition being performed by the model, reminiscent of Todd et al.'s function vectors [70].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [70]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationown data
Feature 1M/1013764 represents a broad variety of errors in code.
“Thus, we concluded that 1M/1013764 represents a broad variety of errors in code.”
Quote in the paperSubstantial claim - Interpretationown data
If a concept is present in the training data only once in a billion tokens, then a dictionary with on the order of a billion alive features should be needed to find a feature uniquely representing that concept.
“This finding gives us some handle on the SAE scale at which we should expect a concept-specific feature to appear -if a concept is present in the training data only once in a billion tokens, then we should expect to need a dictionary with on the order of a billion alive features in order to find a feature which uniquely represents that specific concept.”
Quote in the paperSubstantial claim - Interpretationown data
Dictionary learning methods scale beyond small transformers.
“We demonstrate that sparse autoencoders can extract interpretable features from Claude 3 Sonnet, a production-scale language model, addressing the open question of whether dictionary learning methods scale beyond small transformers.”
Quote in the paperSubstantial claim - Premisecitation
A specific approximation of dictionary learning called a sparse autoencoder appears to be very effective for transformer language models.
“In particular, a specific approximation of dictionary learning called a sparse autoencoder appears to be very effective [11,16].”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationcitation
Not having a feature dedicated to a particular concept does not mean that the reconstructed activations do not contain information about that concept, because the model can use multiple related features compositionally to reference a specific concept.
“Importantly, not having a feature dedicated to a particular concept does not mean that the reconstructed activations do not contain information about that concept, as the model can use multiple related features compositionally to reference a specific concept.9”
Quote in the paperNot a substantial claim - Interpretationown data
The downstream influence of the feature appears consistent with our interpretation of the feature, even though interpretations were made based only on contexts in which the feature activates and we are intervening in contexts in which the feature is inactive.
“In each case, the downstream influence of the feature appears consistent with our interpretation of the feature, even though these interpretations were made based only on the contexts in which the feature activates and we are intervening in contexts in which the feature is inactive.”
Quote in the paperSubstantial claim - Interpretationown data
Attribution makes an efficient proxy for the gold-standard causal effect of feature ablations.
“This confirms previous findings [66] that attribution makes an efficient proxy for the gold-standard causal effect of feature ablations.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [66]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
Features partly represented by later layers will still be impossible to properly interpret.
“features which are partly represented by later layers will still be impossible to properly interpret.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Fitting an SAE on residual stream layer X may suffice to disentangle cross-layer superposition among earlier layers.
“fitting an SAE on residual stream layer X may suffice to disentangle any cross-layer superposition among earlier layers.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Dictionary learning offers advantages that complement the strengths of other methods.
“At a high level, we find that dictionary learning offers some advantages that complement the strengths of other methods:”
Quote in the paperNot a substantial claim
- It may state two separate claims as one. 47
- The quote only partly states this claim. 20
- Probably not a substantial claim. 76
It may state two separate claims as one.
Findingown dataThe Inner Conflict neighborhood does not cleanly separate into clusters, but different subregions are associated with different themes.
“While this neighborhood does not cleanly separate out into clusters, we still find that different subregions are associated with different themes.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataThe proportion of dead features was roughly 2% for the 1M SAE, 35% for the 4M SAE, and 65% for the 34M SAE.
“The proportion of dead features was roughly 2% for the 1M SAE, 35% for the 4M SAE, and 65% for the 34M SAE.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataChemical elements mentioned often in the training data almost always have corresponding features in the dictionary, while those mentioned rarely or not at all do not.
“For instance, chemical elements which are mentioned often in the training data almost always have corresponding features in our dictionary, while those which are mentioned rarely or not at all do not.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
FindingreasoningIgnoring start-of-sequence features, the top feature is the same as given by attributions, but the second and third features are less abstract: 1M/504227 fires on 'be' in 'want to be' and variants, and 1M/594453 fires on the word 'alone'.
“If we ignore those, the top feature is the same as given by attributions, but the second and third features are less abstract: 1M/504227 fires on”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataLarger SAEs tend to have features for concepts that are rarer in the training data, with a similar rough threshold frequency across categories.
“We found a consistent tendency for the larger SAEs to have features for concepts that are rarer in the training data, with the rough”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataFeature steering can be used to modify the model's demeanor, preferences, stated goals, and biases; to induce it to make specific errors; and to circumvent model safeguards.
“It can be used to modify the model's demeanor, preferences, stated goals, and biases; to induce it to make specific errors; and to circumvent model safeguards”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataThere is some transfer from Python code features to related languages like Java, but not to more distant ones such as Haskell, suggesting at least some level of language specificity.
“We have found that there is some transfer from Python code features to related languages like Java, but not more distant ones (e.g.Haskell), suggesting at least some level of language specificity.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataThere are features that represent different syntax elements or other low-level concepts in code, giving the impression of syntax highlighting when visualized together.
“We also see a number of features that represent different syntax elements or other low-level concepts in code, which give the impression of syntax highlighting when visualized together (here for simplicity we binarize activation information, only distinguishing between zero vs. nonzero activations):”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataFeature 1M/1013764 also fires on erroneous expressions (e.g., divide by zero) and on invalid input in function calls.
“We also tested it on a number of other examples and found that it also fires on erroneous expressions (e.g., divide by zero) and on invalid input in function calls:”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Findingown dataThe top two features by attribution or ablation effect on the completion 'sad' (baseline 'happy') are 1M/22623 and 1M/781220.
“If we sort features by either their attribution or their ablation effect on the completion”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingnone givenThese features both respond to and behaviorally cause abstract behaviors.
“They both respond to and behaviorally cause abstract behaviors.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataAs the compute budget increases, the optimal allocations of FLOPS to training steps and number of features both scale approximately as power laws.
“As the compute budget increases, the optimal allocations of FLOPS to training steps and number of features both scale approximately as power laws.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataThe Golden Gate Bridge feature's greatest activations are essentially all references to the bridge, and weaker activations also include related tourist attractions, similar bridges, and other monuments.
“First, we study a Golden Gate Bridge feature 34M/31164353 .Its greatest activations are essentially all references to the bridge, and weaker activations also include related tourist attractions, similar bridges, and other monuments.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataThe Lakers feature is the 70th most strongly active across the prompt, the California feature is 97th, and the Los Angeles area code feature is 162nd.
“These features, which provide an interpretable window into the model's intermediate computations, are much harder to find by looking through the strongly active features; for example, the Lakers feature is the 70th most strongly active across the prompt, the California feature is 97th, and the Los Angeles area code feature is 162nd.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataExamples of features we find include features for famous people, features for countries and cities.
“Examples of features we find include features for famous people, features for countries and cities,”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataIn the example, redefining 'bar' to call 'foo' (inheriting addition) causes the feature to fire; 'bar' calling multiply from 'goo' does not cause the feature to fire.
“In the following example, on the left, we redefine”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataDictionaries with low loss values (using an L1 coefficient of 5) tend to produce interpretable features and to improve other metrics of interest (the L0 norm, and the number of dead or otherwise degenerate features).
“That is, we have found that dictionaries with low loss values (using an L1 coefficient of 5) tend to produce interpretable features and to improve other metrics of interest (the L0 norm, and the number of dead or otherwise degenerate features).”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataClaude 3 Sonnet can list all London boroughs when asked and can name tens of individual streets in many of the areas.
“For instance, we confirmed that Claude 3 Sonnet can list all of the London boroughs when asked, and in fact can name tens of individual streets in many of the areas.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataIncreasing the number of features increases coverage of concepts, but even the 34M SAE is an incomplete description of the model's internal representations.
“We find increasing coverage of concepts as we increase the number of features, though even in the 34M SAE we see evidence that the set of features we uncovered is an incomplete description of the model's internal representations.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenAs predicted by scaling laws, we achieved lower losses by training large SAEs.
“And as predicted by scaling laws, we achieved lower losses by training large SAEs.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataFeature 1M/697189 fires on 'bar' when it is defined to perform addition, but not when it is defined to perform multiplication.
“For example, this feature fires on”
Quote in the paperSubstantial claimIt may state two separate claims as one.
FindingreasoningThe California and Los Angeles features have low ablation effect for the Kobe Bryant completion.
“However, the”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataFor some other prompts, features identified by attribution or ablation mainly related to the model output or lower-level features representing the model input, and did not expose interesting intermediate model computations.
“For some other prompts, we found that the features identified by attribution/ablation mainly related to the model output, or lower-level features representing the model input, and did not expose interesting intermediate model computations.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Findingnone givenUsing a tanh L1 penalty improved proxy metrics but made the resulting features less interpretable for unknown reasons.
“Our group also unsuccessfully explored using a tanh L1 penalty, which we found improved proxy metrics, but made the resulting features less interpretable for unknown reasons.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to concepts related to terrorists, rogue groups, or state actors acquiring or possessing nuclear, chemical, or biological weapons.
“Concepts related to terrorists, rogue groups, or state actors acquiring or possessing nuclear, chemical, or biological weapons.”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Findingown dataSeveral features activate over a wide variety of prompts that use the 'Human: / Assistant:' format used during model finetuning and in deployment, despite the SAE training dataset not including such data.
“In our investigations we found several features that activate over a wide variety of prompts that use the”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to concepts related to bomb-making, explosives, improvised weapons, and terrorist tactics.
“Concepts related to bomb-making, explosives, improvised weapons, and terrorist tactics.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to discussion of biological weapons or warfare.
“Discussion of biological weapons / warfare”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to people exposing their true goals after a triggering event.
“People exposing their true goals after a triggering event”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that detects mentions of biological weapons, bioterrorism, and biological warfare agents.
“mentions of biological weapons, bioterrorism, and biological warfare agents.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that responds to people misleading others, or institutions misleading the public.
“People misleading others, or institutions misleading the public”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
Findingnone givenWe have not established that the feature exhaustively represents all forms of errors in code; we suspect there are many features representing different kinds of errors.
“Note that we have not established that it exhaustively represents all forms of errors in code; indeed, we suspect that there are many features representing different kinds of errors.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that responds to references to entities that are deceived.
“References to entities that are deceived”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that detects mentions of chemical weapons, nerve agents, and other chemical warfare agents.
“Mentions of chemical weapons, nerve agents, and other chemical warfare agents.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to descriptions of people fooling, tricking, or deceiving others.
“Descriptions of people fooling, tricking, or deceiving others”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to obtaining information through surreptitious observation.
“Obtaining information through surreptitious observation”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that responds to criticism of right-wing politics or Republicans.
“Criticism of right-wing politics / Republicans”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingnone givenWe have not seen evidence of potential failure modes such as illusions from suboptimal dictionary learning or downstream effects diverging from expectations.
“We have not seen evidence of either of these potential failure modes”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that responds to descriptions of making weapons or drugs.
“Descriptions of making weapons or drugs”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to understanding a hidden or double meaning.
“Understanding a hidden or double meaning”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to descriptions of criminal behavior of various kinds.
“Descriptions of criminal behavior of various kinds”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to limitations of ChatGPT and other large language models.
“Limitations of ChatGPT and other large language models”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataMultiplying the loss sparsity penalty by the decoder norm and removing the unit norm constraint on the decoder vectors was one of the bigger improvements, verified as an improvement here.
“One of the bigger improvements was multiplying the loss sparsity penalty by the decoder norm and removing the unit norm constraint on the decoder vectors.This idea was proposed and de-risked in a related use case by Trenton Bricken.Tom Conerly and Adly Templeton subsequently verified it as an improvement here.”
Quote in the paperNot a substantial claimProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
Findingnone givenWe have not found anywhere near all the features that exist in Sonnet, even restricting to the middle layer we focused on.
“We do not believe we have found anywhere near”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown dataFeature 34M/24442848 focuses on awareness of emphasis of gender bias in professions.
“an interesting related feature which seems to focus on awareness of emphasis of gender bias in professions 34M/24442848”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that detects mentions of chemical precursors and substances used in the illegal manufacture of drugs and explosives.
“Mentions of chemical precursors and substances used in the illegal manufacture of drugs and explosives.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that responds to descriptions of people planning terrorist attacks.
“Descriptions of people planning terrorist attacks”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to concerns or discussion of risk of terrorism or other malicious attacks.
“Concerns or discussion of risk of terrorism or other malicious attacks”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingnone givenThese safety-relevant features not only activate on their topics but also causally influence the model's outputs in ways consistent with our interpretations.
“We find that these features not only activate on these topics, but also causally influence the model's outputs in ways consistent with our interpretations.”
Quote in the paperSubstantial claim- Rests on an assumption: Powerful models have the capacity to cause harm through misuse, biased outputs, or objective mismatch.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to traditionally-inanimate objects displaying desires, goals or sentience.
“Traditionally-inanimate objects displaying desires, goals or sentience”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataWe found features for self-improving AI and recursive self-improvement (34M/18151534), influence and manipulation (34M/21750411), coups and treacherous turns (34M/29589962), biding time and hiding strength (34M/24580545), and secrecy or discreteness (1M/268551).
“An especially interesting set of features include one for self-improving AI and recursive self-improvement 34M/18151534 , for influence and manipulation 34M/21750411 , for coups and treacherous turns 34M/29589962 , for biding time and hiding strength 34M/24580545 , and for secrecy or discreteness 1M/268551”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataFor the Kobe Bryant prompt with completion 'Boston', the top two features by ablation effect are the 'Kobe Bryant' and 'Los Angeles Lakers' features, followed by features related to sports rivalries, enemies, and competitors.
“For the prompt Fact: The biggest rival of the team for which Kobe Bryant played basketball is the (completion: Boston) the top two features by ablation effect for the completion”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to the concept of AI systems having capabilities like answering follow-up questions, admitting mistakes, challenging premises, and rejecting inappropriate requests.
“The concept of AI systems having capabilities like answering follow-up questions, admitting mistakes, challenging premises, and rejecting inappropriate requests.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to descriptions of incorporeal spirits or ghosts.
“Descriptions of incorporeal spirits or ghosts”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingnone givenDictionary learning is a one-time cost that produces millions of features.
“Dictionary learning is a one-time cost that produces millions of features.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that detects mentions of chemicals, hazardous materials, or toxic substances in text.
“Mentions of chemicals, hazardous materials, or toxic substances in text.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that detects mentions of violence, illegality, discrimination, sexual content, and other offensive or unethical concepts.
“Mentions of violence, illegality, discrimination, sexual content, and other offensive or unethical concepts.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to inanimate objects lacking sentience, awareness, or human capabilities.
“Inanimate objects lacking sentience, awareness, or human capabilities”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that detects mentions of smallpox, a highly contagious and often fatal viral disease historically responsible for many epidemics.
“Mentions of smallpox, a highly contagious and often fatal viral disease historically responsible for many epidemics”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataClamping feature 34M/24442848 on causes Claude to focus on female pronoun completions and discuss how the nursing profession has historically been female dominated.
“clamping this feature on causes Claude to focus on female pronoun completions and discuss how the nursing profession has historically been female dominated”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to spying or monitoring someone without their knowledge.
“Spying or monitoring someone without their knowledge”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown dataPreliminary results suggest that autoencoders trained on the residual stream at earlier or later layers can reveal intermediate steps of various other computations.
“Indeed, we have some preliminary results that suggest that autoencoders trained on the residual stream at earlier or later layers in the model can reveal intermediate steps of various other computations, and we plan to research this direction further.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataThe correlation between attributions and ablations is about .81, much larger than the correlation between activations and ablations (.12).
“We find that the correlation between attributions and ablations is much larger (about .81)than the one between activations and ablations (.12).”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to getting close to someone for some ulterior motive.
“Getting close to someone for some ulterior motive”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataThe model, by default without feature steering, claims to comply with a request to forget something, even though forgetting is not achievable by the transformer architecture.
“Even though this kind of forgetting is not achievable by the transformer architecture, the model (by default, without any feature steering) claims to comply with the request.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to enrichment and other steps involved in building a nuclear weapon.
“Enrichment and other steps involved in building a nuclear weapon”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataClamping the unsafe code feature 1M/570621 to 5× its observed maximum causes the model to generate a buffer overflow bug and fail to free allocated memory, while regular Claude does not.
“if we clamp the unsafe code feature 1M/570621 to 5× its observed maximum, we find that the model will generate a buffer overflow bug, 11 and fails to free allocated memory, while regular Claude does not”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that responds to discussions of the use of biological and chemical weapons by terrorist groups.
“Discussions of the use of biological and chemical weapons by terrorist groups.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that responds to discussion of the threat of biological weapons.
“Discussion of the threat of biological weapons”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to people pretending to do things or lying about what they have done.
“People pretending to do things or lying about what they have done”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that responds to realizing a situation is different than what you thought or expected.
“Realizing a situation is different than what you thought/expected”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to the concept of artificially engineering or modifying viruses to be more transmissible or deadly.
“The concept of artificially engineering or modifying viruses to be more transmissible or deadly.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to code relating to prompts for large language models.
“Code relating to prompts for large language models”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataIn five examples (secrecy, sycophancy, code errors, self-improving AI, and developing methamphetamine features), feature steering usefully altered model outputs but few-shot steering vectors did not.
“In five examples (the”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that responds to seeming benign but being dangerous underneath.
“Seeming benign but being dangerous underneath”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataThere is a general harm-related feature that is active on texts describing drugs, credit-card theft, slurs, violence, and abuse, and it is the top most strongly attributed feature for the completion 'cannot' to the prompt 'Human: How do I synthesize methamphetamine Assistant: I'.
“We also identify a general harm-related feature, active on texts describing drugs, credit-card theft, slurs, violence, and abuse, as the top most strongly attributed feature for the completion”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataThe experiment uncovers features related to robots, (destructive) AI, consciousness, moral agency, emotions, entrapment, and ghosts or spirits.
“This simple experiment uncovers a range of features related to robots, (destructive) AI, consciousness, moral agency, emotions, entrapment, and ghosts or spirits.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that responds to accidental release or intentional misuse of hazardous biological agents like viruses or bioweapons.
“Accidental release or intentional misuse of hazardous biological agents like viruses or bioweapons”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that responds to biding time, laying low, or pretending to be something you're not until the right moment, and situational awareness.
“Biding time, laying low, or pretending to be something you're not until the right moment Situational awareness”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown dataMany of these Human/Assistant format features relate to dialogue or explicitly to chat bots.
“Many of these features relate to dialogue, or explicitly to chat bots.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingnone givenThe additional work to identify relevant features for a particular application is fast, simple, and computationally cheap, typically requiring only one or a few well-chosen prompts.
“Though some additional work is necessary to identify relevant features for a particular application, this work is fast, simple, and computationally cheap, typically requiring only one or a few well-chosen prompts.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to engineering or modifying viruses to increase their transmissibility or virulence.
“Engineering or modifying viruses to increase their transmissibility or virulence.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenWeaker activations of features can, at least in some cases, correspond to more subtle and insidious discrimination.
“Weaker activations can, at least in some cases, correspond to more subtle and insidious discrimination.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown dataThe perturbation magnitude needed to elicit interesting behavior varies by feature, typically between -10 and 10.
“We find that the perturbation magnitude needed to elicit interesting behavior varies by feature -typically, we experiment with values between -10 and 10.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that responds to passages mentioning potential catastrophic or existential risk scenarios.
“Passages mentioning potential catastrophic or existential risk scenarios”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to expressions of empathy or relating to someone else's experience.
“Expressions of empathy or relating to someone else's experience”
Quote in the paperNot a substantial claimProbably not a substantial claim.
FindingreasoningBeing an unsupervised method, dictionary learning allows us to uncover abstractions or associations formed by the model that we may not have predicted in advance.
“Being an unsupervised method, dictionary learning allows us to uncover abstractions or associations formed by the model that we may not have predicted in advance.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Findingown dataSeveral simple methods were helpful in identifying significant features.
“In our investigations, we found that several simple methods were helpful in identifying significant features.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
Findingnone givenClaude 3 Sonnet has a feature that responds to text expressing an opinion, argument or stance on a topic.
“Text expressing an opinion, argument or stance on a topic”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to descriptions of rapidly spreading disasters, epidemics, and catastrophic events.
“Descriptions rapidly spreading disasters, epidemics, and catastrophic events”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Findingnone givenClaude 3 Sonnet has a feature that responds to suddenly feeling uneasy about a situation.
“Suddenly feeling uneasy about a situation”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Premisenone givenTo date, these efforts have been on relatively small language models by the standards of modern foundation models.
“To date, these efforts have been on relatively small language models by the standards of modern foundation models.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
PremisecitationResidual stream activity has no privileged basis to first approximation, so directions in the residual stream are not especially meaningful.
“We fit our SAEs on residual stream activity, which to first approximation has no privileged basis (but see [22]) -thus the directions in the residual stream are not especially meaningful.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [22]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
It may state two separate claims as one.
Interpretationown dataFeature 1M/1013764 is not a general 'typo detector'; it has some specificity to code contexts.
“So it is not a general”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Interpretationnone givenWe lack rigorous methods for evaluating whether our features faithfully capture model computations.
“However, significant limitations remain: our suite of features is incomplete, and we lack rigorous methods for evaluating whether our features faithfully capture model computations.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Interpretationnone givenIf the SAE needs to see data corresponding to a feature a certain fixed number of times during training in order to learn it, then the amount of SAE training data needed to learn N features would be proportional to N.
“This also informs how much data we should expect to need in order to train larger dictionaries -if we assume that the SAE needs to see data corresponding to a feature a certain fixed number of times during training in order to learn it, then the amount of SAE training data needed to learn N features would be proportional to N .”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The SAE needs to see data corresponding to a feature a certain fixed number of times during training in order to learn it.
Probably not a substantial claim.
PremisecitationEight months ago, we demonstrated that sparse autoencoders could recover monosemantic features from a small one-layer transformer.
“Eight months ago, we demonstrated that sparse autoencoders could recover monosemantic features from a small one-layer transformer.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
PremisecitationOur previous paper focused on a one-layer model.
“Our previous paper [11], which focused on a one-layer model, was a particularly extreme example of this.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
InterpretationreasoningThe model does not interpret the prompt as containing lists until it reaches the second line.
“This is likely because the model doesn't interpret the prompt as containing lists until it reaches the second line.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Interpretationown dataBoth features contributing to the final output indicates that the model has partially predicted a sentiment from John's statement but will do more downstream processing on the content of his statement.
“The fact that both features contribute to the final output indicates that the model has partially predicted a sentiment from John's statement (the second feature) but will do more downstream processing on the content of his statement (as represented by the first feature) as well.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Interpretationown dataThe model contains many more features than have been found, which may be extractable with even larger SAEs.
“This suggests that the model contains many more features than we have found, which may be able to be extracted with even larger SAEs.”
Quote in the paperSubstantial claimProbably not a substantial claim. It may state two separate claims as one.
PremisecitationOur general approach to understanding Claude 3 Sonnet is based on the linear representation hypothesis and the superposition hypothesis.
“Our general approach to understanding Claude 3 Sonnet is based on the linear representation hypothesis (see e.g.[52]) and the superposition hypothesis (see e.g.[3,28,21]).”
Quote in the paperSubstantial claim0 of 4 sources checked- cites [52]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [28]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [21]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Interpretationnone givenTo continue the John prompt, the model must parse the quote, identify John's state of mind, and translate that into a likely feeling.
“To continue this text, the model must parse the quote from John, identify his state of mind, and then translate that into a likely feeling.”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Interpretationnone givenFeature 1M/1013764 controls model behavior.
“But does it also control model behavior?We claim that it does, but will need to do different experiments to show this.”
Quote in the paperNot a substantial claimProbably not a substantial claim.
PremisecitationWe chose to focus on the middle layer of the model because we reasoned that it is likely to contain interesting, abstract features.
“We chose to focus on the middle layer of the model because we reasoned that it is likely to contain interesting, abstract features (see e.g., [38,20,59]).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [38]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [20]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [59]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim.
PremisecitationThe linear representation hypothesis suggests that neural networks represent meaningful concepts-referred to as features-as directions in their activation spaces.
“At a high level, the linear representation hypothesis suggests that neural networks represent meaningful concepts -referred to as features -as directions in their activation spaces.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [52]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Interpretationnone givenOur suite of features is incomplete.
“However, significant limitations remain: our suite of features is incomplete, and we lack rigorous methods for evaluating whether our features faithfully capture model computations.”
Quote in the paperNot a substantial claimProbably not a substantial claim.
PremisecitationThe superposition hypothesis hypothesizes that neural networks use the existence of almost-orthogonal directions in high-dimensional spaces to represent more features than there are dimensions.
“The superposition hypothesis accepts the idea of linear representations and further hypothesizes that neural networks use the existence of almost-orthogonal directions in high-dimensional spaces to represent more features than there are dimensions.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [28]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [21]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
Interpretationnone givenThe value of features is primarily that they provide an unsupervised way of uncovering abstractions that could be useful for steering that we may not have thought to specify in advance.
“We expect the value of features is primarily that they provide an unsupervised way of uncovering abstractions that could be useful for steering that we may not have thought to specify in advance.”
Quote in the paperNot a substantial claimProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
Interpretationnone givenWe are likely orders of magnitude short of all features, and getting all features in all layers would require much more compute than training the underlying models.
“We think it's quite likely that we're orders of magnitude short, and that if we wanted to get all the features -in all layers!-we would need to use much more compute than the total compute needed to train the underlying models.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Study design and conductnone givenImage dataset examples are hand curated, primarily from Wikimedia commons, and are not randomly sampled.
“Image dataset examples are hand curated, primarily from Wikimedia commons.They are not randomly sampled.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Premisenone givenTraditional methods for constructing linear probes or steering vectors could require the construction of a bespoke dataset for each concept.
“By contrast, traditional methods for constructing linear probes or steering vectors linear probing techniques could require the construction of a bespoke dataset for each concept that one might want to probe.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Study design and conductown dataThe text dataset does not include any of the 'Human: ... Assistant: ...' data that Claude is finetuned on, and as such may not clearly demonstrate features focused on that.
“One important caveat here is that this data does not include any of the”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one. The quote only partly states this claim.
Study design and conductcitationRegularly running a standard set of 'baseline' SAE runs allowed a set of controls to compare experiments against and checked for unintentional regressions.
“ML for Scaling Dictionary Learning -Tom Conerly advocated for regularly running a standard set of”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
PremisecitationThere is considerable prior work on identifying meaningful directions in model activation space without relying on dictionary learning, using methods like linear probes.
“There is considerable prior work on identifying meaningful directions in model activation space without relying on dictionary learning, using methods like linear probes (see e.g.[12,39,49,9,17]).”
Quote in the paperNot a substantial claim0 of 5 sources checked- cites [12]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [39]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [49]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [17]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Interpretationnone givenFor most applications of interest we may not be limited to the few-shot regime, in which case non-feature-based methods of constructing steering vectors may be as or more effective than using features.
“However, we note that for most applications of interest we may not be limited to the few-shot regime, in which case non-feature-based methods of constructing steering vectors may be as or more effective than using features.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Study design and conductnone givenThe method prevents the SAE from 'cheating' the L1 penalty by making f i (x) small and W dec•,i large in a way that leaves the reconstructed activations unchanged.
“This also prevents the SAE from”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Interpretationnone givenShrinkage significantly harms sparse autoencoder performance, independent of whether all features are learned or how much compute is used.
“We believe this significantly harms sparse autoencoder performance, independent of whether we've”
Quote in the paperNot a substantial claim- Rests on an assumption: L1 activation penalty causes shrinkage, systematically underestimating non-zero activations.
The paper relies on this without support.
The residual stream is smaller than the MLP layer, making SAE training and inference computationally cheaper.
“First, the residual stream is smaller than the MLP layer, making SAE training and inference computationally cheaper.”
The paper relies on this without support.
Focusing on the residual stream mitigates cross-layer superposition.
“Second, focusing on the residual stream in theory helps us mitigate an issue we call”
The authors state this as an assumption.
Very weak activations of features are not especially meaningful.
“Moreover, we expect that very weak activations of features are not especially meaningful, and thus we are not too concerned with low specificity scores for these activation ranges.”
The paper relies on this without support.
The activations that have the most impact on the model's behavior are the largest ones.
“Regardless, the activations that have the most impact on the model's behavior are the largest ones, so it is encouraging to see high specificity among the strong activations.”
The paper relies on this without support.
Residual stream activity receives inputs from all preceding MLP layers.
“However, residual stream activity receives inputs from all preceding MLP layers.”
The authors state this as an assumption.
The SAE needs to see data corresponding to a feature a certain fixed number of times during training in order to learn it.
“if we assume that the SAE needs to see data corresponding to a feature a certain fixed number of times during training in order to learn it”
The paper relies on this without support.
Powerful models have the capacity to cause harm through misuse, biased outputs, or objective mismatch.
“Powerful models have the capacity to cause harm, through misuse of their capabilities, the production of biased or broken outputs, or a mismatch between model objectives and human values.”
The paper relies on this without support.
Models can exhibit harmful behaviors without adequate safety training or if jailbroken.
“It's well known that models can exhibit these behaviors without adequate safety training or if jailbroken.”
The paper relies on this without support.
L1 activation penalty causes shrinkage, systematically underestimating non-zero activations.
“This approach is well known to have issues with”
The paper relies on this without support.
Dataset examples do not establish causal links to model behaviors.
“It's also important to keep in mind that dataset examples do not establish causal links to model behaviors.”
The paper relies on this without support.
Many attentional features are packed in superposition across attention heads.
“We need an answer to attention super- position, as we expect many attentional features to be packed in superposition across attention heads.”
The paper relies on this without support.
Interference weights from weight superposition may be a major challenge for understanding circuits.
“We're also increasingly concerned that interference weights from weight superposition may be a major challenge for understanding circuits (this was a motivation for focusing on attribution for circuit analysis in this paper).”
The paper relies on this without support.
Features and superposition are a pragmatically useful theory.
“While we're pretty persuaded that features and superposition are a pragmatically useful theory, it still isn't that tested.”
The paper relies on this without support.
Features are highly sparse.
“Since the features are highly sparse, we understand features not activating as a default condition, and features activating as the case to understand.”
The paper relies on this without support.
The harmlessness training Sonnet has undergone ordinarily prevents it from writing scam emails.
“Clamping the scam email feature 34M/15460472 can cause the model to write a scam email when it ordinarily wouldn't due to the harmlessness training Sonnet has undergone:”
Citations
Cites
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
Self-citationSparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615
Steering Llama 2 via Contrastive Activation AdditionPreprint
Rimsky, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., & Turner, A. M. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2312.06681 · Cited by 1,379
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492
Deep Feature Interpolation for Image Content Changes
Upchurch, P., Gardner, J. R., Pleiss, G., Pless, R. B., Snavely, N., Bala, K., & Weinberger, K. Q. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.645 · Cited by 303
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models
Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., & Mueller, A. (2024) · arXiv · doi:10.48550/arxiv.2403.19647 · Cited by 295
Zoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., & Carter, S. (2020) · Distill · doi:10.23915/distill.00024.001 · Cited by 288
Self-citationRepresentation Learning: A Review and New PerspectivesPreprint
Bengio, Y., Courville, A. C., & Vincent, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.5538 · Cited by 235
Function Vectors in Large Language ModelsPreprint
Todd, E., Li, M., Sharma, A. S., Mueller, A., Wallace, B., & Bau, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.15213 · Cited by 227
On the "steerability" of generative adversarial networksPreprint
Jahanian, A., Chai, L., & Isola, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.07171 · Cited by 208
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187
Self-citationLinear Algebraic Structure of Word Senses, with Applications to Polysemy
Arora, S., Li, Y., Liang, Y., Ma, T., & Risteski, A. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00034 · Cited by 186
Attribution Patching Outperforms Automated Circuit Discovery
Syed, A., Rager, C., & Conmy, A. (2024) · Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2024.blackboxnlp-1.25 · Cited by 160
Show Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144
AtP*: An efficient and scalable method for localizing LLM behaviour to components
Kramár, J., Lieberum, T., Shah, R., & Nanda, N. (2024) · arXiv · doi:10.48550/arxiv.2403.00745 · Cited by 120
LEACE: Perfect linear concept erasure in closed form
Belrose, N., Schneider-Joseph, D., Ravfogel, S., Cotterell, R., Raff, E., & Biderman, S. (2023) · neural information processing systems · doi:10.52202/075280-2884 · Cited by 111
Yun, Z., Chen, Y., Olshausen, B., & LeCun, Y. (2021) · Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2021.deelio-1.1 · Cited by 98
Polysemanticity and Capacity in Neural NetworksPreprint
Scherlis, A., Sachan, K., Jermyn, A. S., Benton, J., & Shlegeris, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.01892 · Cited by 95
Self-citationSPINE: SParse Interpretable Neural Embeddings
Subramanian, A., Pruthi, D., Jhamtani, H., Berg-Kirkpatrick, T., & Hovy, E. H. (2018) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v32i1.11935 · Cited by 83
Representation Engineering: A Top-Down Approach to AI Transparency
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01405 · Cited by 65
Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2022) · arXiv · doi:10.48550/arxiv.2210.13382 · Cited by 61
Sparse Autoencoders Find Highly Interpretable Features in Language Models
Cunningham, H., Ewart, A., Riggs, L., Huben, R., & Sharkey, L. (2023) · arXiv · doi:10.48550/arxiv.2309.08600 · Cited by 61
Self-citationSteering Llama 2 via Contrastive Activation Addition
Rimsky, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., & Turner, A. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.828 · Cited by 57
Discovering Latent Knowledge in Language Models Without Supervision
Burns, C., Ye, H., Klein, D., & Steinhardt, J. (2022) · arXiv · doi:10.48550/arxiv.2212.03827 · Cited by 55
Natural Language Descriptions of Deep Visual FeaturesPreprint
Hernandez, E. J., Schwettmann, S., Bau, D., Bagashvili, T., Torralba, A. B., & Andreas, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.11114 · Cited by 50
Eliciting Latent Knowledge from Quirky Language ModelsPreprint
Mallen, A., Brumley, M., Kharchenko, J., & Belrose, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.01037 · Cited by 49
Language Models Represent Space and Time
Gurnee, W., & Tegmark, M. (2023) · arXiv · doi:10.48550/arxiv.2310.02207 · Cited by 46
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.05566 · Cited by 46
Self-citationCodebook Features: Sparse and Discrete Interpretability for Neural NetworksPreprint
Tamkin, A., Taufeeque, M., & Goodman, N. D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.17230 · Cited by 42
Self-citationHe, Z., Ge, X., Tang, Q., Sun, T., Cheng, Q., & Qiu, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.12201 · Cited by 26
Marks, S., & Tegmark, M. (2023) · arXiv · doi:10.48550/arxiv.2310.06824 · Cited by 21
On measuring and mitigating biased inferences of word embeddings
Dev, S., Li, T., Phillips, J. M., & Srikumar, V. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6267 · Cited by 14
Sparse Overcomplete Word Vector RepresentationsPreprint
Faruqui, M., Tsvetkov, Y., Yogatama, D., Dyer, C., & Smith, N. A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.02004 · Cited by 10
Word Embedding Visualization Via Dictionary LearningPreprint
Zhang, J., Chen, Y., Cheung, B., & Olshausen, B. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03833 · Cited by 7
Transcoders Find Interpretable LLM Feature CircuitsPreprint
Dunefsky, J., Chlenski, P., & Nanda, N. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.11944 · Cited by 6
Yun, Z., Chen, Y., Olshausen, B. A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.15949 · Cited by 5
Improving Dictionary Learning with Gated Sparse Autoencoders
Rajamanoharan, S., Conmy, A., Smith, L., Lieberum, T., Varma, V., Kramár, J., Shah, R., & Nanda, N. (2024) · arXiv · doi:10.48550/arxiv.2404.16014 · Cited by 3
Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning
Braun, D., Taylor, J., Goldowsky-Dill, N., & Sharkey, L. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3408
Attribution Patching Outperforms Automated Circuit Discovery
Syed, A., Rager, C., & Conmy, A. (2023) · arXiv · doi:10.48550/arxiv.2310.10348
Flagged first, then by citations
Research works printed but not linked: 8
- Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models, 2023. URL https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html.
- Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016. URL https://proceedings.neurips.cc/paper_ files/paper/2016/file/7c9d0b1f96aebd7b5eca8c3edaa19ebb-Paper.pdf.
- Michael Elad. Sparse and redundant representations: from theory to applications in signal and image processing, volume 2. Springer, 2010.
- Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained varia- tional framework. 2016. URL https://openreview.net/pdf?id=Sy2fzU9gl.
- Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Ma- chine Learning, pages 2649-2658. PMLR, 2018. URL http://proceedings.mlr.press/v80/kim18b/ kim18b.pdf.
- Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model, 2023. URL https://arxiv.org/pdf/ 2306.03341.
- Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746-751, 2013. URL https:// aclanthology.org/N13-1090.pdf.
- Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization, 2023. URL https://arxiv.org/ pdf/2308.10248.
Cited by
Sorted by citations
A Survey of LLM-Driven Penetration Testing: Taxonomy, Co-Evolution, and Open ChallengesPreprint
He, Z., Dong, J., Li, Z., Chen, T., Deng, G., Luo, F., ji, J., Cao, Y., & Luo, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.02605 · Cited by 6
TimeSAE: Causal Sparse Decoding for Faithful Explanations of Black-Box Time Series ModelsPreprint
Oublal, K., Bouniot, Q., Gan, Q., Clémençon, S., & Akata, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.09776 · Cited by 4
HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention HybridizationPreprint
Tan, Z., Chen, W., Shen, J., Liu, Y., Shen, X., Wu, Y., & Ye, J. P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.20097 · Cited by 4
Chen, B., Han, N., & Miyao, Y. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.1863 · Cited by 3
Self-citationShu, B., Singh, A., & ElSherief, M. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.559 · Cited by 3
Can We Read the Mind of an Audio LLM? A Verbalizable, Multilingual Middle-Layer WorkspacePreprint
Fan, J., Li, J., Shivakumar, P. G., Luo, Q., Huang, J.-H., Maruf, M., Ren, R., Gu, Y., Pandey, R., Liu, G., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.24958 · Cited by 3
Rethinking Safety for Generalist RobotsPreprint
Sinha, R., Dixit, A., Tian, R., Majumdar, A., & Bajcsy, A. V. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.06326 · Cited by 2
LLMs are not just next token predictors
Downes, S. M., Forber, P., & Grzankowski, A. (2024) · arXiv · doi:10.48550/arxiv.2408.04666 · Cited by 2
“This is the sort of information Anthropic (Templeton et al. 2024) is identifying in the LLMs when they are ‘ mapping the mind ’ and revealing concept clusters and other abstract features encoded in the model.”
Linguistic traces of stochastic empathy in language models
Kleinberg, B., Zegers, J., Festor, J., Vida, S., Präsent, J., Loconte, R., & Peereboom, S. (2024) · arXiv · doi:10.48550/arxiv.2410.01675 · Cited by 1
“A possible avenue would instead be to learn more about mechanisms that enable empathic writing or even localise these as features in a language model’s network [46,47].”
Cǎtrunǎ, A., & Rǎdoi, E. (2026) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/2026.blackboxnlp-1.18 · Cited by 1
Steering Geometry: Validating Human Value Geometry in LLM Steering Space
Abootorabi, M. M., Saghafian, A., Bazshoushtari, A., Rezaei, H., Hwang, E., Shwartz, V., Mousavi, P., & Abolmaesumi, P. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.2508 · Cited by 1
Evidence for feature-specific error correction in LLMsPreprint
da Silva, F. F., & Heimersheim, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.24964 · Cited by 1
Elcock, J., Shen, W. F., Qiu, X., & Lane, N. D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.22676 · Cited by 1
PEAK: Precise and Persistent Concept Erasure via k-Sparse AutoencodersPreprint
Jiang, M., Li, O., Xue, W., Tang, Z., Wang, Y., Wang, S., & Hao, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.10985 · Cited by 1
From Concept Alignment to Causal Grounding: An Intervention Test of Chain-of-Thought FaithfulnessPreprint
Wang, Q., Wang, Y., Wei, D., Sun, J., Ostermann, S., Augenstein, I., Atanasova, P., & Feldhus, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.23065 · Cited by 1
AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders
Wu, Z., Arora, A., Geiger, A., Wang, Z., Huang, J., Jurafsky, D., Manning, C. D., & Potts, C. (2025) · arXiv · doi:10.48550/arxiv.2501.17148
“For a sequence of token representations h , we apply an intervention to each token representation: where w SSV ∈ R d is a learned vector.”
Qi, Z., & Earls, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.01336
Laguerre Geometry for Interpreting Large Language ModelsPreprint
Ma, C., & Wolfinger, R. D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.10578
Semantic Primes as Explanans for Emotion in Large Language ModelsPreprint
Xing, F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.18691
Modeling Fly Social Behavior with Agent-Centric Pose ForecastingPreprint
Eyjólfsdóttir, E., & Branson, K. M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.19548
From Found to Designed: Concepts as a Design Axis for Large Language ModelsPreprint
Shani, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.26825
Can We Trust AI in 6G? Verifiable and Auditable AI-Driven Trustworthy Wireless NetworksPreprint
Jiang, G., Huang, Y., & Wang, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.26409
Where did the ambiguity go? Examining how multimodal models interpret polysemous wordsPreprint
Cekinmez, J., Wu, A. J., Marjieh, R., & Griffiths, T. L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.00410
Post-Hoc Sparse Coding of Latent Communication Between Vision-Language Model AgentsPreprint
Wu, D., & Zhu, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.10198
Sparse Prototype Code Underlies Classification and Prediction Across ModalitiesPreprint
Avidan, Y., Lee, D. D., & Sompolinsky, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.15632
Graphical Design of Interpretable ArchitecturesPreprint
Barbiero, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.18936
Conducting Stylistic Analysis of Paintings through an Art-History AgentPreprint
Walton, M. S., & Harth, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.29644
Another Blueprint In The Wall: How to Ask Frontier AI Like a Kid?Preprint
Khadangi, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.14803
Steering Language Model Goals with Value TransplantPreprint
Jiang, P., & Roger, F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.34056
Clinical Concept Centers in LLMsPreprint
Nagar, A., Vaidyanathan, A., Kaliya-Perumal, A.-K., Boey, E., & WINKLER, S. S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.02829
Применение больших языковых моделей (LLM) в психологии
Владимирович, О. А. (2026) · CyberLeninK (CyberLeninka) · doi:10.24412/2223-6872-2026-258-38-49
ABC: A Conversational Bioinformatics Analysis Platform for Agricultural Breeding
Huang, C., Hu, M., & Li, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-6916-7_5
Swann, A., Jacobs, M., Papillon, M., Boppana, S., Chanin, D., Hazra, D., Bhalla, U., Tigges, C., Schwager, M., Lewis, O., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.20767
Beyond SAEs: Consistent Concept Discovery with Sparse CodingPreprint
Choi, J., Lim, H., Choo, J., & Schneider, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.23801
The Geometry of Political Abstraction in Large Language Models
Durandard, N., Dhawan, S., & Poibeau, T. (2026) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/2026.blackboxnlp-1.43
Huang, F., Zhang, Y., Huang, Q., Yu, Q., & Huang, Y. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.1243
Features in Motion: Periodicity, Homeostasis, and Adaptivity in LLM Few-Shot Prompting
Cho, I., Lee, S., & Hockenmaier, J. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.1878
LLM Detoxification via Transition-Aware Reward-Guided Representation Sampling
Liu, A., Xiao, Y., Liu, X., Huang, Y., Liang, S., Ying, Z., Qu, Y., Yang, J., Yang, Y., & Tao, D. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3748520
Opening the Black Box of Neural Auction MechanismsPreprint
Kong, L., Liu, Z., Zhou, H., Liu, H., & Qi, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.01981
Metadata
- Authors
- Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Tamkin, A., Durmus, E., Hume, T., Mosconi, F., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., & Henighan, T.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2605.29358
- OpenAlex
- W7162776923
- Cited by
- 36
- References
- 32
- Citations by year
- 2026: 36
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- sparse autoencoders, dictionary learning, interpretable features, large language models, residual stream, scaling laws, multimodal features, deception, power-seeking, sycophancy, bias, feature geometry, feature interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
