The DOI matches its Crossref record; the references it cites are not checked yet.
A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
Open accessAbstract
Large Language Models (LLMs) have transformed natural language processing, yet their internal mechanisms remain largely opaque.Recently, mechanistic interpretability has attracted significant attention from the research community as a means to understand the inner workings of LLMs.Among various mechanistic interpretability approaches, Sparse Autoencoders (SAEs) have emerged as a promising method due to their ability to disentangle the complex, superimposed features within LLMs into more interpretable components.This paper presents a comprehensive survey of SAEs for interpreting and understanding the internal workings of LLMs.Our major contributions include: (1) exploring the technical framework of SAEs, covering basic architecture, design improvements, and effective training strategies;(2) examining different approaches to explaining SAE features, categorized into input-based and output-based explanation methods; (3) discussing evaluation methods for assessing SAE performance, covering both structural and functional metrics; and (4) investigating real-world applications of SAEs in understanding and manipulating LLM behaviors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Sparse Autoencoders (SAEs) decompose a model's hidden activations into a set of sparse and interpretable latent features, trained via reconstruction objectives with sparsity regularization to isolate semantically meaningful directions.
“Sparse Autoencoders (SAEs) decompose a model's hidden activations into a set of sparse and interpretable latent features, trained via reconstruction objectives with sparsity regularization to isolate semantically meaningful directions (Shu et al., 2025).”
From Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancy (2026) · cites this paper as Shu et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 62
- Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. Linear algebraic struc- ture of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495.
- Kola Ayonrinde. 2024. Adaptive sparse allocation with mutual choice & feature choice sparse autoencoders. arXiv preprint arXiv:2411.02124. · arXiv
- Leonard Bereska and Efstratios Gavves. 2024a. Mech- anistic interpretability for ai safety-a review. arXiv preprint arXiv:2404.14082. · arXiv
- Leonard Bereska and Stratis Gavves. 2024b. Mecha- nistic interpretability for ai safety-a review. Transac- tions on Machine Learning Research.
- Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language mod- els across training and scaling. In International Conference on Machine Learning, pages 2397-2430.
- PMLR.
- Steven Bills, Nick Cammarata, Dan Moss- ing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Language mod- els can explain neurons in language models. https://openaipublic.blob.core.windows. net/neuron-explainer/paper/index.html.
- Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey. 2025. Identifying functionally im- portant features with end-to-end sparse dictionary learning. Advances in Neural Information Process- ing Systems, 37:107286-107325.
- Bart Bussmann, Patrick Leask, and Neel Nanda. 2024. Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410. · arXiv
- Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy. 2024. Improving steering vectors by target- ing sparse autoencoder features. arXiv preprint arXiv:2411.02193. · arXiv
- Thomas M Cover. 1999. Elements of information theory. John Wiley & Sons.
- Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600. · arXiv
- Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated con- volutional networks. In International conference on machine learning, pages 933-941. PMLR.
- DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Eoin Farrell, Yeu-Tong Lau, and Arthur Conmy. 2024. Applying sparse autoencoders to unlearn knowledge in language models. In Neurips Safe Generative AI Workshop 2024.
- Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah Smith. 2015. Sparse overcom- plete word vector representations. arXiv preprint arXiv:1506.02004. · arXiv
- Javier Ferrando, Oscar Balcells Obeso, Senthooran Ra- jamanoharan, and Neel Nanda. 2025. Do i know this entity? knowledge awareness and hallucinations in language models. In The Thirteenth International Conference on Learning Representations.
- Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. 2024. A primer on the in- ner workings of transformer-based language models. arXiv preprint arXiv:2405.00208. · arXiv
- Alex Foote, Neel Nanda, Esben Kran, Ioannis Konstas, Shay Cohen, and Fazl Barez. 2023. Neuron to graph: Interpreting language model neurons at scale. arXiv preprint arXiv:2305.19911. · arXiv
- Andrey Galichin, Alexey Dontsov, Polina Druzhinina, Anton Razzhigaev, Oleg Y Rogov, Elena Tutubalina, and Ivan Oseledets. 2025. I have covered all the bases here: Interpreting reasoning features in large language models via sparse autoencoders. arXiv preprint arXiv:2503.18878. · arXiv
- Jack Gallifant, Shan Chen, Kuleen Sasse, Hugo Aerts, Thomas Hartvigsen, and Danielle S Bitterman. 2025. Sparse autoencoder features for classifications and transferability. arXiv preprint arXiv:2502.11367. · arXiv
- Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. 2019. Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907.12009. · arXiv
- Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2025. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations.
- Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Arya- man Arora, Zhengxuan Wu, Noah Goodman, Christo- pher Potts, et al. 2023. Causal abstraction: A the- oretical foundation for mechanistic interpretability. arXiv preprint arXiv:2301.04709. · arXiv
- Davide Ghilardi, Federico Belotti, and Marco Moli- nari. 2024. Efficient training of sparse autoencoders for large language models via layer groups. arXiv preprint arXiv:2410.21508. · arXiv
- Mario Giulianelli, Jacqueline Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. arXiv preprint arXiv:1808.08079. · arXiv
- Karol Gregor and Yann LeCun. 2010. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference on international conference on machine learning, pages 399-406.
- Clément Guerner, Anej Svete, Tianyu Liu, Alexan- der Warstadt, and Ryan Cotterell. 2023. A geo- metric notion of causal probing. arXiv preprint arXiv:2307.15054. · arXiv
- Yoav Gur-Arieh, Roy Mayan, Chen Agassy, Atticus Geiger, and Mor Geva. 2025. Enhancing automated interpretability with output-centric feature descrip- tions. arXiv preprint arXiv:2501.08319. · arXiv
- Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, et al. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. arXiv preprint arXiv:2410.20526. · arXiv
- Zirui He, Haiyan Zhao, Yiran Qiao, Fan Yang, Ali Payani, Jing Ma, and Mengnan Du. 2025. Saif: A sparse autoencoder framework for interpreting and steering instruction following of language models. arXiv preprint arXiv:2502.11356. · arXiv
- Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. 2018. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230. · arXiv
- Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, et al. 2025. Ex- ploring concept depth: How large language models acquire knowledge and concept at different layers? In Proceedings of the 31st International Conference on Computational Linguistics, pages 558-573.
- Subhash Kantamneni, Joshua Engels, Senthooran Ra- jamanoharan, Max Tegmark, and Neel Nanda. 2025. Are sparse autoencoders useful? a case study in sparse probing. Preprint, arXiv:2502.16681. · arXiv
- Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018. In- terpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In In- ternational conference on machine learning, pages 2668-2677. PMLR.
- Hyunjik Kim and Andriy Mnih. 2018. Disentangling by factorising. In International conference on machine learning, pages 2649-2658. PMLR.
- Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathe- matical statistics, 22(1):79-86.
- Justin Lee, Tuomas Oikarinen, Arjun Chatha, Keng-Chi Chang, Yilan Chen, and Tsui-Wei Weng. 2023. The importance of prompt tuning for automated neuron explanations. arXiv preprint arXiv:2310.06200. · arXiv
- Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147. · arXiv
- Francesco Locatello, Stefan Bauer, Mario Lucic, Gun- nar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. 2019. Challenging common as- sumptions in the unsupervised learning of disentan- gled representations. In international conference on machine learning, pages 4114-4124. PMLR.
- Christos Louizos, Max Welling, and Diederik P Kingma. 2017. Learning sparse neural networks through l_0 regularization. arXiv preprint arXiv:1712.01312. · arXiv
- Luke Marks, Alasdair Paren, David Krueger, and Fazl Barez. 2024. Enhancing neural network interpretabil- ity with feature-aligned sparse autoencoders. arXiv preprint arXiv:2411.01220. · arXiv
- Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associa- tions in gpt. Advances in neural information process- ing systems, 35:17359-17372.
- Abhinav Menon, Manish Shrivastava, David Krueger, and Ekdeep Singh Lubana. 2024. Analyzing (in) abilities of saes via formal languages. arXiv preprint arXiv:2410.11767. · arXiv
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013a. Distributed representa- tions of words and phrases and their compositionality. Advances in neural information processing systems, 26.
- Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the as- sociation for computational linguistics: Human lan- guage technologies, pages 746-751.
- Anish Mudide, Joshua Engels, Eric J Michaud, Max Tegmark, and Christian Schroeder de Witt. 2024. Ef- ficient dictionary learning with switch sparse autoen- coders. arXiv preprint arXiv:2410.08201. · arXiv
- Aashiq Muhamed, Mona Diab, and Virginia Smith. 2024. Decoding dark matter: Specialized sparse autoencoders for interpreting rare concepts in foun- dation models. arXiv preprint arXiv:2411.00743. · arXiv
- Neel Nanda, Andrew Lee, and Martin Wattenberg. 2023. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941. · arXiv
- Andrew Ng et al. 2011. Sparse autoencoder. CS294A Lecture notes, 72(2011):1-19.
- Anh Nguyen, Jason Yosinski, and Jeff Clune. 2019. Un- derstanding neural networks via feature visualization: A survey. Explainable AI: interpreting, explaining and visualizing deep learning, pages 55-76.
- Angus Nicolson, Lisa Schut, J Alison Noble, and Yarin Gal. 2024. Explaining explainability: Under- standing concept activation vectors. arXiv preprint arXiv:2404.03713. · arXiv
- Bruno A Olshausen and David J Field. 1997. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311- 3325.
- Charles O'Neill, Christine Ye, Kartheik Iyer, and John F Wu. 2024. Disentangling dense embed- dings with sparse autoencoders. arXiv preprint arXiv:2408.00657. · arXiv
- Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388- 12401.
- Xuansheng Wu, Wenhao Yu, Xiaoming Zhai, and Ning- hao Liu. 2025a. Self-regularization with latent space explanations for controllable llm-based classification. arXiv preprint arXiv:2502.14133. · arXiv
- Xuansheng Wu, Jiayi Yuan, Wenlin Yao, Xiaoming Zhai, and Ninghao Liu. 2025b. Interpreting and steering llms with mutual information-based expla- nations on sparse autoencoders. arXiv preprint arXiv:2502.15576. · arXiv
- Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. 2025c. Axbench: Steering llms? even simple baselines outperform sparse autoencoders. Preprint, arXiv:2501.17148. · arXiv
- Xianjun Yang, Shaoliang Nie, Lijuan Liu, Suchin Guru- rangan, Ujjwal Karn, Rui Hou, Madian Khabsa, and Yuning Mao. 2025. Diversity-driven data selection for language model tuning through sparse autoen- coder. arXiv preprint arXiv:2502.14050. · arXiv
- Qingyu Yin, Chak Tou Leong, Hongbo Zhang, Minjun Zhu, Hanqi Yan, Qiang Zhang, Yulan He, Wenjie Li, Jun Wang, Yue Zhang, et al. 2024. Direct preference optimization using sparse feature-level constraints. arXiv preprint arXiv:2411.07618. · arXiv
- Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2024a. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15(2):1-38.
- Haiyan Zhao, Fan Yang, Bo Shen, Himabindu Lakkaraju, and Mengnan Du. 2024b. Towards uncovering how large language model works: An explainability perspective. arXiv preprint arXiv:2402.10688. · arXiv
- Haiyan Zhao, Heng Zhao, Bo Shen, Ali Payani, Fan Yang, and Mengnan Du. 2025. Beyond single con- cept vector: Modeling concept subspace in llms with gaussian distribution. The Thirteenth International Conference on Learning Representations.
Cited by
Sorted by citations
Arous, I., Chehbouni, K., Cheng, Z., & Dossou, B. F. P. (2025) · Springer eBooks · doi:10.1007/978-981-97-8440-0_85-1 · Cited by 5
Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancy
Feng, Z., Chen, Z., Ma, J., Po, Y. T., Chersoni, E., & Li, B. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1126 · Cited by 2
Orlov, A. V., Makus, Y. V., Ashniev, G. A., Orlova, N. N., & Nikitin, P. I. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.03.04.709491 · Cited by 2
Selective LLM Unlearning via SAE-Based Token Importance Score
Kim, S. Y., Lim, J. H., Lee, D. W., Min, S. H., Kim, I. J., Jo, M. I., Shin, J. Y., & Kim, W. G. (2025) · International Symposium on Networks, Computers and Communications (ISNCC) · doi:10.1109/isncc66965.2025.11250473
ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Models
Li, H., Wen, Z., Jiang, Q., Li, C., Wu, Y., Yang, Y.-C. B., Wang, Y., Huang, X., Zhu, M., & Chen, W. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3634806
Reversing the Death of Hypertext: Mechanistic Interpretability for LLM Navigation
Zheng, W., He, Y., & Li, J. (2026) · ACM SIGWEB Newsletter · doi:10.1145/3801080.3801084
Marzouny, N. H., & Dziedzic, R. (2026) · World Environmental and Water Resources Congress (EWRI) · doi:10.1061/9780784486931.067
Arous, I., Chehbouni, K., Cheng, Z., & Dossou, B. F. P. (2026) · Springer eBooks · doi:10.1007/978-981-95-3658-0_85
Can LLMs Acquire Human-Like World Models?
Teoharie, C. (2026) · Philosophies · doi:10.3390/philosophies11030094
Towards principled knowledge editing methods for large language model reasoning
Yao, Y., Qin, J., Zhang, N., Xu, H., Zhu, Y., Yu, Z., Wang, M., Tang, Y., Gu, J.-C., Deng, S., et al. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01276-y
A survey of theory-grounded interpretability for deep neural networks
Khalid, M. U., Alharbi, A. A. K., Bashir, S., & Rehman, S. U. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01061-2
Language Organization in Multilingual LLMs Through SAE Feature Suppression
Presa, J. P. C., Santos, W. F., & de Oliveira, S. S. T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39892-5_23
Dugăeșescu, A., & Florea, A. M. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100315
Interpretability of EEG Models via Sparse Autoencoders
Farooq, F., & Wang, M. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100337
Metadata
- Authors
- Shu, D., Wu, X., Zhao, H., Rai, D., Yao, Z., Liu, N., & Du, M.
- Affiliations
- Northwestern University (US)
- University of Georgia (US)
- New Jersey Institute of Technology (US)
- George Mason University (US)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2025
- Published
- 2025
- Pages
- 1690-1712
- OpenAlex
- W4416036098
- Cited by
- 13
- References
- 0
- Citations by year
- 2026: 10 · 2025: 3
- FWCI
- 12.39
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- Funders
- National Science Foundation (2223768, 2507128, 2310261)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- sparse autoencoders, mechanistic interpretability, large language models, feature disentanglement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
