CoChat
Sign Up
PreprintCited by 75Topic ModelingNatural Language Processing TechniquesSpeech Recognition and Synthesis

Augmenting Self-attention with Persistent Memory

Open access
Sukhbaatar, Sainbayar, Grave, Edouard, Lample, Guillaume, Jegou, Herve, Joulin, Armand·2019·arXiv
DOI 10.48550/arxiv.1907.01470arXiv 1907.01470
Open this paper in CoChat — free accountOpen PDF

Abstract

Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feed-forward layer and a self-attention layer. The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers. Building upon this intuition, we propose a new model that solely consists of attention layers. More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer. Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer. Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1907.01470

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Replacing the ReLU in fully connected layers with Softmax can result in attention-like weights, in which weights are data-independent.

    “Sukhbaatar, Grave, et al. (2019) showed that replacing the ReLU in fully connected layers with Softmax can results in an attention-like weights, in which weights are data-independent:”

    From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Sukhbaatar, Grave, et al. (2019)

    Section 4.1 (p. 4) says replacing ReLU with Softmax and removing biases makes the feedforward sublayer interpretable as attention. But it says the resulting attention weight is computed using the input x_t, so that weight is not data-independent. Section 4.2 (p. 4) describes the persistent key-value vectors—not the attention weights—as unconditioned on the input. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Persistent memory weights are expected to have the same functionality as FFN weights, so using them in the first part of the sequence leads to input-independent attention weights.

    “The persistent memory weights are expected to have the same functionality, meaning that using them in the first part of the sequence leads to having input-independent attention weights (Sukhbaatar, Grave, et al. 2019).”

    From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Sukhbaatar, Grave, et al. 2019

    Section 4.2 (p. 4) says persistent vectors play a role similar to feedforward-layer weights and are not conditioned on the input. But Section 4.1 (p. 4) describes the attention weight as computed using the input vector, so input-independent persistent vectors do not imply input-independent attention weights. The cited passages also do not establish the claimed consequence for the first part of a sequence. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • FFNs in a transformer can be interpreted as persistent memories that augment self-attention layers and provide means to represent information that persists across input sequences.

    “Persistent memories Sukhbaatar et al. (2019) shows that FFNs in a transformer can be interpreted as persistent memories that augment the self-attention layers and provide means to represent information that persists across input sequences.”

    From Memory Mosaics (2024) · cites this paper as Sukhbaatar et al. (2019)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 40 · Cited by 75

Research works printed but not linked: 40

  1. [1][1] Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence , 2019. -
  2. [2][2] Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In ICLR , 2019. -
  3. [3][3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR , 2015. -
  4. [4][4] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research , 3(Feb):1137-1155, 2003. -
  5. [5][5] Yoshua Bengio, Jean-Sébastien Senécal, et al. Quick training of probabilistic neural nets by importance sampling. In AISTATS , pages 1-9, 2003. -
  6. [6][6] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019. - · arXiv
  7. [8][8] Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 , 2019. - · arXiv
  8. [9][9] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In ICML , 2017. -
  9. [11][11] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12(Jul):2121-2159, 2011. -
  10. [12][12] Joshua T Goodman. A bit of progress in language modeling. Computer Speech & Language , 15(4):403-434, 2001. -
  11. [13][13] Edouard Grave, Armand Joulin, Moustapha Cissé, and Hervé Jégou. Efficient softmax approximation for gpus. In ICML , 2017. -
  12. [14][14] Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous cache. In ICLR , 2017. -
  13. [15][15] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401 , 2014. - · arXiv
  14. [16][16] David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In ICLR , 2017. -
  15. [18][18] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8): 1735-1780, 1997. -
  16. [19][19] Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In ICLR , 2017. -
  17. [20][20] Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation , 6(2):181-214, 1994. -
  18. [21][21] Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410 , 2016. - · arXiv
  19. [22][22] Ben Krause, Iain Murray, Steve Renals, and Liang Lu. Multiplicative LSTM for sequence modelling. In ICLR (Workshop) , 2017. -
  20. [23][23] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. - · arXiv
  21. [25][25] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR , 2017. -
  22. [26][26] Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An analysis of neural language modeling at multiple scales. arXiv preprint arXiv:1803.08240 , 2018. - · arXiv
  23. [28][28] Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. In EMNLP , 2016. -
  24. [29][29] Frederic Morin and Yoshua Bengio. Hierarchical probabilistic neural network language model. In AISTATS , 2005. -
  25. [30][30] Asier Mujika, Florian Meier, and Angelika Steger. Fast-slow recurrent neural networks. In NIPS , pages 5915-5924, 2017. -
  26. [31][31] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In ICML , 2013. -
  27. [32][32] Ofir Press and Lior Wolf. Using the output embedding to improve language models. In EACL (2) , 2017. -
  28. [33][33] Jack W. Rae, Chris Dyer, Peter Dayan, and Timothy P. Lillicrap. Fast parametric learning with activation memorization. In ICML , 2018. -
  29. [34][34] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL (1) , 2016. -
  30. [35][35] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In NAACL-HLT (2) , 2018. -
  31. [36][36] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR , 2017. -
  32. [37][37] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research , 15(1):1929-1958, 2014. -
  33. [38][38] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In NIPS , 2015. -
  34. [39][39] Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. In ACL , 2019. -
  35. [40][40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , 2017. -
  36. [41][41] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NIPS , 2015. -
  37. [42][42] Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In ICLR , 2019. -
  38. [43][43] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML , 2015. -
  39. [44][44] Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 , 2014. - · arXiv
  40. [45][45] Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway networks. In ICML , 2017.

Authors
Sukhbaatar, S., Grave, E., Lample, G., Jegou, H., & Joulin, A.
Affiliations
Meta (United Kingdom) (GB)
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1907.01470
arXiv
1907.01470
OpenAlex
W2955227499
Cited by
75
References
40
Citations by year
2025: 2 · 2024: 3 · 2023: 19 · 2022: 13 · 2021: 20 · 2020: 13 · 2019: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
self-attention, persistent memory, long-range dependencies, language modeling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review