CoChat
Sign Up
PreprintCited by 3

The DOI matches its DataCite record.

Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations

Csordás, Róbert, Potts, Christopher, Manning, Christopher D., Geiger, Atticus·2024·arXiv
DOI 10.48550/arxiv.2408.10920arXiv 2408.10920
Open this paper in CoChat — free account

Abstract

The Linear Representation Hypothesis (LRH) states that neural networks learn to encode concepts as directions in activation space, and a strong version of the LRH states that models learn only such encodings. In this paper, we present a counterexample to this strong LRH: when trained to repeat an input token sequence, gated recurrent neural networks (RNNs) learn to represent the token at each position with a particular order of magnitude, rather than a direction. These representations have layered features that are impossible to locate in distinct linear subspaces. To show this, we train interventions to predict and manipulate tokens by learning the scaling factor corresponding to each sequence position. These interventions indicate that the smallest RNNs find only this magnitude-based solution, while larger RNNs have linear representations. These findings strongly indicate that interpretability research should not be confined by the LRH.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2408.10920

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The strong version of the linear representation hypothesis is false for some models.

    “Some works have shown that the strong version is false for some models (Black et al., 2022;Csordás et al., 2024).”

    From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Csordás et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 41

  1. Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. A latent variable model approach to PMI-based word embeddings. Transac- tions of the Association for Computational Linguis- tics, 4:385-399. · DOI · Paper page
  2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. Preprint arXiv:1607.06450. · arXiv
  3. Nora Belrose, David Schneider-Joseph, Shauli Ravfo- gel, Ryan Cotterell, Edward Raff, and Stella Bider- man. 2023. LEACE: perfect linear concept erasure in closed form. In Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.
  4. Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computa- tion. Preprint arXiv:1308.3432. · arXiv
  5. Kyunghyun Cho, Bart van Merriënboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724- 1734, Doha, Qatar. Association for Computational Linguistics. · DOI · Paper page
  6. Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. 2021. Are neural nets modular? In- specting functional modularity through differentiable weight masks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  7. Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Con- ference on Machine Learning (ICML).
  8. Xander Davies, Max Nadeau, Nikhil Prakash, Tamar Rott Shaham, and David Bau. 2023. Discover- ing variable binding circuitry with desiderata. CoRR, abs/2307.03637. · DOI · Paper page
  9. Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. 2020. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 3243-3255, On- line. Association for Computational Linguistics. · DOI · Paper page
  10. Jeffrey L. Elman. 1990. Finding structure in time. Cog- nitive Science, 14(2):179-211. · DOI · Paper page
  11. Jeffrey L. Elman. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7(2):195-225. · DOI · Paper page
  12. Joshua Engels, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark. 2024. Not all language model features are linear. CoRR, abs/2405.14860. · DOI
  13. Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021. Causal analysis of syntactic agreement mechanisms in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1828-1843, Online. Association for Computational Linguistics. · DOI
  14. Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Arya- man Arora, Zhengxuan Wu, Noah Goodman, Christo- pher Potts, and Thomas Icard. 2024a. Causal ab- straction: A theoretical foundation for mechanistic interpretability. Preprint, arXiv:2301.04709. · arXiv
  15. Atticus Geiger, Hanson Lu, Thomas Icard, and Christo- pher Potts. 2021. Causal abstractions of neural net- works. In Advances in Neural Information Process- ing Systems 34: Annual Conference on Neural In- formation Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 9574-9586.
  16. Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Net- works for NLP, pages 163-173, Online. Association for Computational Linguistics. · DOI
  17. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. 2024b. Find- ing alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, 1-3 April 2024, Los Ange- les, California, USA, volume 236 of Proceedings of Machine Learning Research, pages 160-187.
  18. PMLR.
  19. Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem H. Zuidema. 2018. Under the hood: Using diagnostic classifiers to in- vestigate and improve how language models track agreement information. In Proceedings of the Work- shop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2018, Brussels, Bel- gium, November 1, 2018, pages 240-248. Associa- tion for Computational Linguistics. · DOI
  20. Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. · arXiv
  21. Albert Gu, Karan Goel, and Christopher Ré. 2021a. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396. · arXiv
  22. Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. 2021b. Com- bining recurrent, convolutional, and continuous-time models with linear state space layers. In Advances in Neural Information Processing Systems, volume 34, pages 572-585.
  23. Clément Guerner, Anej Svete, Tianyu Liu, Alexander Warstadt, and Ryan Cotterell. 2023. A geometric notion of causal probing. CoRR, abs/2307.15054. · DOI
  24. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735- 1780.
  25. Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. 2024. RAVEL: Evalu- ating interpretability methods on disentangling lan- guage model representations. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8669-8687, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
  26. Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2024. Sparse autoen- coders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations.
  27. Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categor- ical reparametrization with gumbel-softmax. In Int. Conf. on Learning Representations (ICLR), Toulon, France.
  28. Omer Levy and Yoav Goldberg. 2014. Linguistic regu- larities in sparse and explicit word representations. In Proceedings of the Eighteenth Conference on Compu- tational Natural Language Learning, pages 171-180, Ann Arbor, Michigan. Association for Computational Linguistics. · DOI
  29. Chris J Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The concrete distribution: A continuous relax- ation of discrete random variables. In Int. Conf. on Learning Representations (ICLR), Toulon, France.
  30. Aleksandar Makelov, George Lange, and Neel Nanda. 2024. Towards principled evaluations of sparse au- toencoders for interpretability and control. Preprint, arXiv:2405.08366. · arXiv
  31. Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2024. Sparse feature circuits: Discovering and editing in- terpretable causal graphs in language models. CoRR, abs/2403.19647. · DOI · Paper page
  32. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746-751, Atlanta, Georgia. Association for Computational Linguistics.
  33. Neel Nanda, Andrew Lee, and Martin Wattenberg. 2023. Emergent linear representations in world models of self-supervised sequence models. In Proceed- ings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, Black- boxNLP@EMNLP 2023, Singapore, December 7, 2023, pages 16-30. Association for Computational Linguistics. · DOI · Paper page
  34. Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. The linear representation hypothesis and the geometry of large language models. CoRR, abs/2311.03658. · DOI · Paper page
  35. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computa- tional Linguistics. · DOI · Paper page
  36. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guard- ing protected attributes by iterative nullspace projec- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7237-7256. Association for Computational Linguistics. · DOI · Paper page
  37. Paul Smolensky. 1986. Neural and conceptual inter- pretation of PDP models. In James L. McClelland, David E. Rumelhart, and the PDP Research Group, editors, Parallel Distributed Processing: Explo- rations in the Microstructure of Cognition: Psycho- logical and Biological Models, volume 2, pages 390- 431. MIT Press.
  38. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Pro- cessing Systems 30, pages 5998-6008.
  39. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stu- art M. Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  40. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Inter- pretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh Inter- national Conference on Learning Representations.
  41. Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christo- pher Potts, and Noah D. Goodman. 2023. Inter- pretability at scale: Identifying causal mechanisms in alpaca. In Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.

Cited by

Sorted by citations

Metadata

Authors
Csordás, R., Potts, C., Manning, C. D., & Geiger, A.
Venue
arXiv
Publisher
arXiv
Published
2024
DOI
10.48550/arxiv.2408.10920
arXiv
2408.10920
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review