The DOI matches its DataCite record; 1 of the 5 references it cites were found in a registry or an index, and 4 are not checked yet.
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
Open accessAbstract
Sparse autoencoders (SAEs) are an unsupervised method for learning a sparse decomposition of a neural network's latent representations into seemingly interpretable features. Despite recent excitement about their potential, research applications outside of industry are limited by the high cost of training a comprehensive suite of SAEs. In this work, we introduce Gemma Scope, an open suite of JumpReLU SAEs trained on all layers and sub-layers of Gemma 2 2B and 9B and select layers of Gemma 2 27B base models. We primarily train SAEs on the Gemma 2 pre-trained models, but additionally release SAEs trained on instruction-tuned Gemma 2 9B for comparison. We evaluate the quality of each SAE on standard metrics and release these results. We hope that by releasing these SAE weights, we can help make more ambitious safety and interpretability research easier for the community. Weights and a tutorial can be found at https://huggingface.co/google/gemma-scope and an interactive demo can be found at https://www.neuronpedia.org/gemma-scope
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 33
- S. Bills, N. Cammarata, D. Mossing, H. Till- man, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons in language mod- els. https://openaipublic.blob. core.windows.net/neuron-explainer/ paper/index.html, 2023.
- A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso. To- wards automated circuit discovery for mech- anistic interpretability, 2023. URL https: //arxiv.org/abs/2304.14997.
- J. Dunefsky, P. Chlenski, and N. Nanda. Transcoders find interpretable llm feature cir- cuits, 2024. URL https://arxiv.org/abs/ 2406.11944.
- J. Engels, I. Liao, E. J. Michaud, W. Gurnee, and M. Tegmark. Not all language model features are linear, 2024. URL https://arxiv.org/ abs/2405.14860. · arXiv
- L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The pile: An 800gb dataset of diverse text for lan- guage modeling, 2020. URL https://arxiv. org/abs/2101.00027.
- L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu. Scaling and evaluating sparse autoencoders, 2024. URL https://arxiv.org/abs/2406. 04093.
- Gemini Team. Gemini: A family of highly ca- pable multimodal models, 2024. URL https: //arxiv.org/abs/2312.11805.
- Gemma Team. Gemma: Open models based on gemini research and technology, 2024a. URL https://arxiv.org/abs/2403.08295. · arXiv
- W. Gurnee, N. Nanda, M. Pauly, K. Har- vey, D. Troitskii, and D. Bertsimas. Find- ing neurons in a haystack: Case studies with sparse probing. Transactions on Ma- chine Learning Research, 2023. ISSN 2835- 8856. URL https://openreview.net/ forum?id=JYs1R9IMJr.
- W. Gurnee, T. Horsley, Z. C. Guo, T. R. Kheirkhah, Q. Sun, W. Hathaway, N. Nanda, and D. Bertsi- mas. Universal neurons in gpt2 language mod- els, 2024. URL https://arxiv.org/abs/ 2401.12181.
- M. Hanna, O. Liu, and A. Variengien. How does gpt-2 compute greater-than?: Interpret- ing mathematical abilities in a pre-trained lan- guage model, 2023. URL https://arxiv. org/abs/2305.00586.
- K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification, 2015. URL https://arxiv.org/abs/1502.01852. · arXiv
- R. Hendel, M. Geva, and A. Globerson. In- context learning creates task vectors. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id= QYvFUlF19n.
- J. Huang, A. Geiger, K. D'Oosterlinck, Z. Wu, and C. Potts. Rigorously assessing natural language explanations of neurons, 2023. URL https: //arxiv.org/abs/2309.10312.
- S. Jain, E. S. Lubana, K. Oksuz, T. Joy, P. H. S. Torr, A. Sanyal, and P. K. Dokania. What makes and breaks safety fine-tuning? a mechanistic study, 2024. URL https://arxiv.org/abs/2407. 10264.
- A. Karvonen, B. Wright, C. Rager, R. Angell, J. Brinkmann, L. R. Smith, C. M. Verdun, D. Bau, and S. Marks. Measuring progress in dictionary learning for language model inter- pretability with board game models. In ICML 2024 Workshop on Mechanistic Interpretabil- ity, 2024. URL https://openreview.net/ forum?id=qzsDKwGJyB.
- D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017. URL https: //arxiv.org/abs/1412.6980.
- C. Kissane, R. Krzyzanowski, J. I. Bloom, A. Conmy, and N. Nanda. Interpreting at- tention layer outputs with sparse autoen- coders, 2024a. URL https://arxiv.org/ abs/2406.17759. · arXiv
- P. Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceed- ings of Machine Translation Summit X: Papers, pages 79-86, Phuket, Thailand, Sept. 13-15 2005. URL https://aclanthology.org/ 2005.mtsummit-papers.
- K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg. Inference-time interven- tion: Eliciting truthful answers from a lan- guage model. In Thirty-seventh Confer- ence on Neural Information Processing Sys- tems, 2023. URL https://openreview. net/forum?id=aLLuYpn83y.
- A. Makelov, G. Lange, and N. Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. In ICLR 2024 Workshop on Secure and Trustwor- thy Large Language Models, 2024. URL https://openreview.net/forum?id= MHIX9H8aYF.
- T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space, 2013. URL https://arxiv. org/abs/1301.3781.
- S. Rajamanoharan, A. Conmy, L. Smith, T. Lieberum, V. Varma, J. Kramár, R. Shah, and N. Nanda. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014, 2024a. · arXiv
- S. Rajamanoharan, T. Lieberum, N. Sonnerat, A. Conmy, V. Varma, J. Kramár, and N. Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, 2024b. URL https://arxiv.org/abs/ 2407.14435.
- D. Saxton, E. Grefenstette, F. Hill, and P. Kohli. Analysing mathematical reasoning abilities of neural models, 2019. URL https://arxiv. org/abs/1904.01557.
- M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language mod- els using model parallelism, 2020. URL https: //arxiv.org/abs/1909.08053.
- A. Stolfo, Y. Belinkov, and M. Sachan. A mech- anistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In The 2023 Conference on Em- pirical Methods in Natural Language Process- ing, 2023. URL https://openreview.net/ forum?id=aB3Hwh4UzP.
- E. Todd, M. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau. Function vectors in large language models. In The Twelfth In- ternational Conference on Learning Represen- tations, 2024. URL https://openreview. net/forum?id=AwyxtyMwaG.
- A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid. Activation addition: Steering language models without optimization, 2024. URL https:// arxiv.org/abs/2308.10248. · arXiv
- K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt- 2 small, 2022. URL https://arxiv.org/ abs/2211.00593. · arXiv
- M. Wattenberg and F. Viégas. Relational composition in neural networks: A gen- tle survey and call to action. In ICML 2024 Workshop on Mechanistic Interpretabil- ity, 2024. URL https://openreview.net/ forum?id=zzCEiUIPk9.
- B. Zhang and R. Sennrich. Root mean square layer normalization, 2019. URL https:// arxiv.org/abs/1910.07467. · arXiv
- D. Ziegler, S. Nix, L. Chan, T. Bauman, P. Schmidt- Nielsen, T. Lin, A. Scherlis, N. Nabeshima, B. Weinstein-Raun, D. de Haas, B. Shlegeris, and N. Thomas. Adversarial training for high- stakes reliability. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, ed- itors, Advances in Neural Information Processing Systems, volume 35, pages 9274-9286.
Cited by
Sorted by citations
Open Problems in Mechanistic Interpretability
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.16496 · Cited by 291
Self-citationEberhardt, S. T., Vehlen, A., Schaffrath, J., Schwartz, B., Baur, T., Schiller, D., Hallmen, T., André, E., & Lutz, W. (2025) · Scientific Reports · doi:10.1038/s41598-025-14923-y · Cited by 23
AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders
Wu, Z., Arora, A., Geiger, A., Wang, Z., Huang, J., Jurafsky, D., Manning, C. D., & Potts, C. (2025) · arXiv · doi:10.48550/arxiv.2501.17148
“We use SAEs from GemmaScope that are trained for these layers.”
InfluentialSAGE: Signal-Amplified Guided Embeddings for Vulnerability Detection
Shan, Z., Qian, X., Xin, J., Xu, M., Zhang, Y., Yang, Z., Wu, H., & Cheng, X. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832141 · Cited by 1
Metadata
- Authors
- Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L., Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., & Nanda, N.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2408.05147
- OpenAlex
- W4402502788
- Cited by
- 2
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- sparse autoencoders, Gemma 2, interpretability, large language model safety, instruction-tuned models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
