CoChat
Sign Up
PreprintCited by 14Topic ModelingFace and Expression RecognitionNeural Networks and Applications

The DOI matches its DataCite record.

The Geometry of Concepts: Sparse Autoencoder Feature Structure

Open access
Li, Yuxiao, Michaud, Eric J., Baek, David D., Engels, Joshua, Sun, Xiaoqing, Tegmark, Max·2024·arXiv
DOI 10.48550/arxiv.2410.19750arXiv 2410.19750PMC PMC12025678
Open this paper in CoChat — free account

Abstract

Sparse autoencoders have recently produced dictionaries of high-dimensional vectors corresponding to the universe of concepts represented by large language models. We find that this concept universe has interesting structure at three levels: 1) The "atomic" small-scale structure contains "crystals" whose faces are parallelograms or trapezoids, generalizing well-known examples such as (man-woman-king-queen). We find that the quality of such parallelograms and associated function vectors improves greatly when projecting out global distractor directions such as word length, which is efficiently done with linear discriminant analysis. 2) The "brain" intermediate-scale structure has significant spatial modularity; for example, math and code features form a "lobe" akin to functional lobes seen in neural fMRI images. We quantify the spatial locality of these lobes with multiple metrics and find that clusters of co-occurring features, at coarse enough scale, also cluster together spatially far more than one would expect if feature geometry were random. 3) The "galaxy" scale large-scale structure of the feature point cloud is not isotropic, but instead has a power law of eigenvalues with steepest slope in middle layers. We also quantify how the clustering entropy depends on the layer.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.19750

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs are capable of modeling the external world through cognitive maps, as indicated by recent research revealing brain-like structures embedded within them.

    “Furthermore, LLMs are capable of modeling the external world through cognitive maps, as indicated by recent research revealing the brain-like structures embedded within them [117].”

    From Understanding World or Predicting Future? A Comprehensive Survey of World Models (2026) · cites this paper as [117]

    Section 4 (“Brain”-Scale: Meso-Scale Modular Structure) investigates whether functionally related features form spatial lobes analogous to functional clustering in animal brains; the Conclusions report significant spatial modularity. The paper does not establish that LLMs model the external world through cognitive maps, so that part of the claim goes beyond its findings. Page numbers were not supplied for these passages. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Li et al. discover brain-like structures of world knowledge embedded in the high-dimensional vectors that represent the universe of concepts in large language models.

    “Furthermore, Li et al. [117] discover brain-like structures of world knowledge embedded in the high-dimensional vectors that represent the universe of concepts in large language models.”

    From Understanding World or Predicting Future? A Comprehensive Survey of World Models (2026) · cites this paper as [117]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 56

  1. [1]1. Hurst, A.; Lerer, A.; Goucher, A.P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; et al. Gpt-4o system card. arXiv 2024, arXiv:2410.21276. · arXiv
  2. [3]3. Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv 2025, arXiv:2501.12948. · arXiv
  3. [4]4. Slattery, P.; Saeri, A.K.; Grundy, E.A.; Graham, J.; Noetel, M.; Uuk, R.; Dao, J.; Pour, S.; Casper, S.; Thompson, N. The ai risk repository: A comprehensive meta-review, database, and taxonomy of risks from artificial intelligence. arXiv 2024, arXiv:2408.12622. · arXiv
  4. [5]5. Sharma, M.; Tong, M.; Korbak, T.; Duvenaud, D.; Askell, A.; Bowman, S.R.; Cheng, N.; Durmus, E.; Hatfield-Dodds, Z.; Johnston, S.R.; et al. Towards understanding sycophancy in language models. arXiv 2023, arXiv:2310.13548. · arXiv
  5. [6]6. Park, P.S.; Goldstein, S.; O'Gara, A.; Chen, M.; Hendrycks, D. AI deception: A survey of examples, risks, and potential solutions. arXiv. arXiv 2023, arXiv:2308.14752. · arXiv
  6. [7]7. Marks, S.; Treutlein, J.; Bricken, T.; Lindsey, J.; Marcus, J.; Mishra-Sharma, S.; Ziegler, D.; Ameisen, E.; Batson, J.; Belonax, T.; et al. Auditing Language Models for Hidden Objectives. arXiv 2024, arXiv:2503.10965. · arXiv
  7. [8]8. Ngo, R.; Chan, L.; Mindermann, S. The alignment problem from a deep learning perspective. arXiv 2022, arXiv:2209.00626. · arXiv
  8. [9]9. Bereska, L.; Gavves, E. Mechanistic Interpretability for AI Safety—A Review. arXiv 2024, arXiv:2404.14082. · arXiv
  9. [10]10. Sharkey, L.; Chughtai, B.; Batson, J.; Lindsey, J.; Wu, J.; Bushnaq, L.; Goldowsky-Dill, N.; Heimersheim, S.; Ortega, A.; Bloom, J.; et al. Open Problems in Mechanistic Interpretability. arXiv 2025, arXiv:2501.16496. · arXiv
  10. [11]11. Huben, R.; Cunningham, H.; Smith, L.R.; Ewart, A.; Sharkey, L. Sparse Autoencoders Find Highly Interpretable Features in Language Models. In Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria, 7–11 May 2024.
  11. [13]13. Templeton, A.; Conerly, T.; Marcus, J.; Lindsey, J.; Bricken, T.; Chen, B.; Pearce, A.; Citro, C.; Ameisen, E.; Jones, A.; et al. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread 2024. Available online: https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html (accessed on 24 March 2025). https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf https://transformer-circuits.pub/2023/monosemantic-features/index.html https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html Entropy 2025, 27, 344 15 of 16
  12. [14]14. Faruqui, M.; Tsvetkov, Y.; Yogatama, D.; Dyer, C.; Smith, N. Sparse overcomplete word vector representations. arXiv 2015, arXiv:1506.02004. · arXiv
  13. [15]15. Zhang, J.; Chen, Y.; Cheung, B.; Olshausen, B.A. Word embedding visualization via dictionary learning. arXiv 2019, arXiv:1910.03833. · arXiv
  14. [16]16. Yun, Z.; Chen, Y.; Olshausen, B.A.; LeCun, Y. Transformer visualization via dictionary learning: Contextualized embedding as a linear superposition of transformer factors. arXiv 2021, arXiv:2103.15949. · arXiv
  15. [17]17. Olshausen, B.A.; Field, D.J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 1996, 381, 607–609.
  16. [18]18. Olshausen, B.A.; Field, D.J. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vis. Res. 1997, 37, 3311–3325.
  17. [20]20. Park, K.; Choe, Y.J.; Veitch, V. The linear representation hypothesis and the geometry of large language models. arXiv 2023, arXiv:2311.03658. · arXiv
  18. [22]22. Engels, J.; Liao, I.; Michaud, E.J.; Gurnee, W.; Tegmark, M. Not All Language Model Features Are Linear. arXiv 2024, arXiv:2405.14860. · arXiv
  19. [23]23. Lieberum, T.; Rajamanoharan, S.; Conmy, A.; Smith, L.; Sonnerat, N.; Varma, V.; Kramár, J.; Dragan, A.; Shah, R.; Nanda, N. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv 2024, arXiv:2408.05147. · arXiv
  20. [24]24. Ansuini, A.; Laio, A.; Macke, J.H.; Zoccolan, D. Intrinsic dimension of data representations in deep neural networks. Adv. Neural Inf. Process. Syst. 2019, 32, 1–15.
  21. [25]25. Chandrasekaran, D.; Mago, V. Evolution of semantic similarity—A survey. ACM Comput. Surv. (CSUR) 2021, 54, 1–37.
  22. [26]26. Watanabe, S. Algebraic Geometry and Statistical Learning Theory; Cambridge University Press: Cambridge, UK, 2009; Volume 25.
  23. [27]27. Rushing, C.; Nanda, N. Explorations of Self-Repair in Language Models. arXiv 2024, arXiv:2402.15390. · arXiv
  24. [28]28. Belrose, N.; Furman, Z.; Smith, L.; Halawi, D.; Ostrovsky, I.; McKinney, L.; Biderman, S.; Steinhardt, J. Eliciting latent predictions from transformers with the tuned lens. arXiv 2023, arXiv:2303.08112. · arXiv
  25. [29]29. Conmy, A.; Mavor-Parker, A.; Lynch, A.; Heimersheim, S.; Garriga-Alonso, A. Towards automated circuit discovery for mechanistic interpretability. Adv. Neural Inf. Process. Syst. 2023, 36, 16318–16352.
  26. [30]30. Park, K.; Choe, Y.J.; Jiang, Y.; Veitch, V. The geometry of categorical and hierarchical concepts in large language models. arXiv 2024, arXiv:2406.01506. · arXiv
  27. [34]34. Drozd, A.; Gladkova, A.; Matsuoka, S. Word embeddings, analogies, and machine learning: Beyond king − man + woman = queen. In Proceedings of the Coling 2016, the 26th International Conference on Computational Linguistics: Technical Papers; The COLING 2016 Organizing Committee: Osaka, Japan, 2016; pp. 3519–3530.
  28. [35]35. Pennington, J.; Socher, R.; Manning, C.D. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP); Association for Computational Linguistics: Doha, Qatar, 2014; pp. 1532–1543.
  29. [36]36. Ma, L.; Zhang, Y. Using Word2Vec to process big text data. In Proceedings of the 2015 IEEE International Conference on Big Data (Big Data), Santa Clara, CA, USA, 29 October–1 November 2015 ; pp. 2895–2897.
  30. [37]37. Nanda, N.; Lee, A.; Wattenberg, M. Emergent linear representations in world models of self-supervised sequence models. arXiv 2023, arXiv:2309.00941. · arXiv
  31. [38]38. Li, K.; Hopkins, A.K.; Bau, D.; Viégas, F.; Pfister, H.; Wattenberg, M. Emergent world representations: Exploring a sequence model trained on a synthetic task. arXiv 2022, arXiv:2210.13382. · arXiv
  32. [39]39. Michaud, E.J.; Liao, I.; Lad, V.; Liu, Z.; Mudide, A.; Loughridge, C.; Guo, Z.C.; Kheirkhah, T.R.; Vukelić, M.; Tegmark, M. Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code. Entropy 2024, 26, 1046. https://transformer-circuits.pub/2022/toy_model/index.html https://transformer-circuits.pub/2022/toy_model/index.html https://transformer-circuits.pub/2024/july-update/index.html#linear-representations https://www.alignmentforum.org/posts/MFBTjb2qf3ziWmzz6/sae-feature-geometry-is-outside-the-superposition-hypothesis https://www.alignmentforum.org/posts/MFBTjb2qf3ziWmzz6/sae-feature-geometry-is-outside-the-superposition-hypothesis https://www.lesswrong.com/posts/tojtPCCRpKLSHBdpn/the-strong-feature-hypothesis-could-be-wrong https://www.lesswrong.com/posts/tojtPCCRpKLSHBdpn/the-strong-feature-hypothesis-could-be-wrong https://www.alignmentforum.org/posts/TMAmHh4DdMr4nCSr5/showing-sae-latents-are-not-atomic-using-meta-saes https://www.alignmentforum.org/posts/TMAmHh4DdMr4nCSr5/showing-sae-latents-are-not-atomic-using-meta-saes Entropy 2025, 27, 344 16 of 16
  33. [40]40. Marks, S.; Tegmark, M. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv 2023, arXiv:2310.06824. · arXiv
  34. [41]41. Gurnee, W.; Tegmark, M. Language models represent space and time. arXiv 2023, arXiv:2310.02207. · arXiv
  35. [42]42. Heinzerling, B.; Inui, K. Monotonic representation of numeric properties in language models. arXiv 2024, arXiv:2403.10381. · arXiv
  36. [43]43. Todd, E.; Li, M.L.; Sharma, A.S.; Mueller, A.; Wallace, B.C.; Bau, D. Function vectors in large language models. arXiv 2023, arXiv:2310.15213. · arXiv
  37. [44]44. Hendel, R.; Geva, M.; Globerson, A. In-context learning creates task vectors. arXiv 2023, arXiv:2310.15916. · arXiv
  38. [46]46. Meng, K.; Bau, D.; Andonian, A.; Belinkov, Y. Locating and editing factual associations in gpt. Adv. Neural Inf. Process. Syst. 2022, 35, 17359–17372.
  39. [47]47. Ahmed, M.; Seraj, R.; Islam, S.M.S. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics 2020, 9, 1295.
  40. [48]48. Xanthopoulos, P.; Pardalos, P.M.; Trafalis, T.B.; Xanthopoulos, P.; Pardalos, P.M.; Trafalis, T.B. Linear discriminant analysis. In Robust Data Mining; Springer: New York, NY 2013; pp. 27–33.
  41. [49]49. Gao, L.; Biderman, S.; Black, S.; Golding, L.; Hoppe, T.; Foster, C.; Phang, J.; He, H.; Thite, A.; Nabeshima, N.; et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv 2020, arXiv:2101.00027. · arXiv
  42. [50]50. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830.
  43. [51]51. Van der Maaten, L.; Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 2008, 9, 2579–2605.
  44. [52]52. Vinh, N.X.; Epps, J.; Bailey, J. Information theoretic measures for clusterings comparison: Is a correction for chance necessary? In Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, QC, Canada, 14–18 June 2009; pp. 1073–1080.
  45. [53]53. Mueller, A.; Brinkmann, J.; Li, M.; Marks, S.; Pal, K.; Prakash, N.; Rager, C.; Sankaranarayanan, A.; Sharma, A.S.; Sun, J.; et al. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. arXiv 2024, arXiv:2408.01416. · arXiv
  46. [55]55. Hoel, E.P.; Albantakis, L.; Tononi, G. Quantifying causal emergence shows that macro can beat micro. Proc. Natl. Acad. Sci. USA 2013, 110, 19790–19795.
  47. [56]56. Kennicutt, R.C., Jr. Star formation in galaxies along the Hubble sequence. Annu. Rev. Astron. Astrophys. 1998, 36, 189–231.
  48. [57]57. Hubble, E.P. Extragalactic Nebulae. Astrophys. J. 1926, 64, 321–369. https://doi.org/10.1086/143018. · DOI
  49. [58]58. Kravtsov, A. Dark matter substructure and dwarf galactic satellites. Adv. Astron. 2010, 2010, 281913.
  50. [59]59. Wishart, J. The generalised product moment distribution in samples from a normal multivariate population. Biometrika 1928, 20, 32–52.
  51. [60]60. Marchenko, V.; Pastur, L.A. Distribution of eigenvalues for some sets of random matrices. Mat. Sb. 1967, 72, 4.
  52. [61]61. Dasarathy, B.V. Nearest Neighbor (NN) Norms: NN Pattern Classification Techniques; IEEE Computer Society Tutorial: Los Alamitos, CA, USA, 1991.
  53. [62]62. Kozachenko, L.F.; Leonenko, N.N. Sample estimate of the entropy of a random vector. Probl. Peredachi Informatsii 1987, 23, 9–16.
  54. [63]63. Jaccard, P. Nouvelles Recherches Sur La Distribution Florale. Bull. De La Société Vaudoise Des Sci. Nat. 1908, 44, 223–270.
  55. [64]64. Dice, L.R. Measures of the Amount of Ecologic Association Between Species. Ecology 1945, 26, 297–302.
  56. [65]65. Yule, G.U. On the Methods of Measuring Association Between Two Attributes. J. R. Stat. Soc. 1912, 75, 579–652. Disclaimer/Publisher's Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. https://www.alignmentforum.org/posts/5FGXmJ3wqgGRcbyH7/extracting-sae-task-features-for-in-context-learning https://www.alignmentforum.org/posts/5FGXmJ3wqgGRcbyH7/extracting-sae-task-features-for-in-context-learning https://transformer-circuits.pub/2023/interpretability-dreams/index.html https://doi.org/10.1086/143018 · DOI

Cited by

Sorted by citations

Metadata

Authors
Li, Y., Michaud, E. J., Baek, D. D., Engels, J., Sun, X., & Tegmark, M.
Affiliations
  • The NSF AI Institute for Artificial Intelligence and Fundamental Interactions (US)
  • Massachusetts Institute of Technology (US)
Corresponding author
Tegmark, Max
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
27
Issue
4
Pages
344-344
ISSN
1099-4300 (ISSN-L)
DOI
10.48550/arxiv.2410.19750
arXiv
2410.19750
PMC
PMC12025678
OpenAlex
W4408919020
Cited by
14
References
0
Citations by year
2026: 6 · 2025: 8
FWCI
5.62
Type
journal article
Language
en
Open access
Licence
Locations
  • Entropy (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (PHY-2 019 786, 2019786, 2141064)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
sparse autoencoders, concept vectors, large language models, feature geometry, linear discriminant analysis, word embeddings

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review