CoChat
Sign Up
PreprintCited by 76Topic ModelingNatural Language Processing TechniquesAdvanced Neural Network Applications

Large Memory Layers with Product Keys

Open access
Lample, Guillaume, Sablayrolles, Alexandre, Ranzato, Marc'Aurelio, Denoyer, Ludovic, Jégou, Hervé·2019·arXiv
DOI 10.48550/arxiv.1907.05242arXiv 1907.05242
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper introduces a structured memory which can be easily integrated into a neural network. The memory is very large by design and significantly increases the capacity of the architecture, by up to a billion parameters with a negligible computational overhead. Its design and access pattern is based on product keys, which enable fast and exact nearest neighbor search. The ability to increase the number of parameters while keeping the same computational budget lets the overall system strike a better trade-off between prediction accuracy and computation efficiency both at training and test time. This memory layer allows us to tackle very large scale language modeling tasks. In our experiments we consider a dataset with up to 30 billion words, and we plug our memory layer in a state-of-the-art transformer-based architecture. In particular, we found that a memory augmented model with only 12 layers outperforms a baseline transformer model with 24 layers, while being twice faster at inference time. We release our code for reproducibility purposes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1907.05242

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Using a shared pool of memory parameters across all memory layers keeps the parameter count the same and maximizes parameter sharing.

    “In contrast to previous work (Lample et al., 2019), we use a shared pool of memory parameters across all memory layers, thus keeping parameter count the same and maximizing parameter sharing.”

    From Memory Layers at Scale (2024) · cites this paper as Lample et al., 2019

    The Training details (p. 6) describe placing multiple memories at different transformer layers, so the citation identifies relevant previous work. The paper says memory heads share values within a memory (Multi-head memory attention, p. 5), but our copy does not establish whether parameters are shared across memory layers. The claimed contrast in cross-layer sharing is therefore not fully supported; keeping the parameter count the same is the citing authors’ description of their own method. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Product-key networks were introduced to make memory lookup more efficient and scalable.

    “Product-key networks (Lample et al., 2019) were introduced to make the memory lookup more efficient and scalable.”

    From Memory Layers at Scale (2024) · cites this paper as Lample et al., 2019

  • Product keys enable efficient top-k lookup on the full key set by first searching smaller half-key sets, saving compute and memory.

    “The top-k lookup on the full set of keys can be efficiently done by searching the much smaller set of half-keys first, saving compute and memory.”

    From Memory Layers at Scale (2024) · cites this paper as Lample et al., 2019

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 44

  1. [1][1] Artem Babenko and Victor Lempitsky. The inverted multi-index. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014. -
  2. [2][2] Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference on Representation Learning, 2019. -
  3. [3][3] Yoshua Bengio, Nicholas L´ eonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013. - · arXiv
  4. [4][4] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. Conference of the International Speech Communication Association, 2014. -
  5. [6][6] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. CoRR, abs/1406.7362, 2014. - · arXiv
  6. [7][7] Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR, 2016. -
  7. [8][8] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in Neural Information Processing Systems, 2015. -
  8. [9][9] Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Conference of the Association for Computational Linguistics, 2019. -
  9. [10][10] Ludovic Denoyer and Patrick Gallinari. Deep sequential neural network. CoRR, abs/1410.0510, 2014. - · arXiv
  10. [12][12] D. Eigen, I. Sutskever, and M. Ranzato. Learning factored representations in a deep mixture of experts. In Workshop at the International Conference on Learning Representations, 2014. -
  11. [13][13] Thomas Gerald, Nicolas Baskiotis, and Ludovic Denoyer. Binary stochastic representations for large multi-class classification. In International Conference on Neural Information Processing, 2017. -
  12. [14][14] Edouard Grave, Moustapha M Cisse, and Armand Joulin. Unbounded cache model for online language modeling with open vocabulary. In Advances in Neural Information Processing Systems, 2017. -
  13. [15][15] Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous cache. In International Conference on Representation Learning, 2017. -
  14. [16][16] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In International Conference on Acoustics, Speech, and Signal Processing, 2013. -
  15. [17][17] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. CoRR, abs/1410.5401, 2014. - · arXiv
  16. [18][18] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision. In Conference on Computer Vision and Pattern Recognition, 2017. -
  17. [19][19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, 2016. -
  18. [20][20] Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. CoRR, abs/1811.06965, 2018. - · arXiv
  19. [21][21] Herv´ e J´ egou, Matthijs Douze, and Cordelia Schmid. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011. -
  20. [22][22] Jeff Johnson, Matthijs Douze, and Herv´ e J´ egou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 2017. -
  21. [23][23] Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In Advances in Neural Information Processing Systems, 2015. -
  22. [24][24] Koray Kavukcuoglu, Marc'Aurelio Ranzato, and Yann LeCun. Fast inference in sparse coding algorithms with applications to object recognition. CoRR, abs/1010.3467, 2010. URL http: //arxiv.org/abs/1010.3467 . - · arXiv
  23. [25][25] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Representation Learning, 2015. -
  24. [26][26] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. Moses: Open source toolkit for statistical machine translation. In Conference of the Association for Computational Linguistics, 2007. -
  25. [27][27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2012. -
  26. [28][28] Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, 2019. -
  27. [29][29] Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In European Conference on Computer Vision, 2018. -
  28. [30][30] Alireza Makhzani and Brendan Frey. K-sparse autoencoders. In International Conference on Representation Learning, 2014. -
  29. [31][31] Alireza Makhzani and Brendan J Frey. Winner-take-all autoencoders. In Advances in Neural Information Processing Systems, 2015. -
  30. [32][32] Marius Muja and David G. Lowe. Scalable nearest neighbor algorithms for high dimensional data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014. -
  31. [33][33] Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over-parametrization in generalization of neural networks. In International Conference on Representation Learning, 2019. -
  32. [35][35] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Neurips Autodiff Workshop, 2017. -
  33. [37][37] Jack Rae, Jonathan J Hunt, Ivo Danihelka, Timothy Harley, Andrew W Senior, Gregory Wayne, Alex Graves, and Timothy Lillicrap. Scaling memory-augmented neural networks with sparse reads and writes. In Advances in Neural Information Processing Systems, 2016. -
  34. [38][38] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, 2016. -
  35. [39][39] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Conference of the Association for Computational Linguistics, 2015. -
  36. [40][40] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In International Conference on Representation Learning, 2017. -
  37. [41][41] Stefano Spigler, Mario Geiger, St´ ephane d'Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under- to over-parametrization affects loss landscape and generalization. CoRR, abs/1810.09665, 2018. - · arXiv
  38. [42][42] Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In Advances in Neural Information Processing Systems, 2015. -
  39. [43][43] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, 2014. -
  40. [44][44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. -
  41. [45][45] Sudheendra Vijayanarasimhan, Jonathon Shlens, Rajat Monga, and Jay Yagnik. Deep networks with large output spaces. In Workshop at the International Conference on Learning Representations, 2015. -
  42. [46][46] Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Representation Learning, 2018. -
  43. [47][47] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In International Conference on Representation Learning, 2015. -
  44. [48][48] Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. In International Conference on Representation Learning, 2016.

Cited by

Sorted by citations

Related

Authors
Lample, G., Sablayrolles, A., Ranzato, M., Denoyer, L., & Jégou, H.
Affiliations
  • Sorbonne Université (FR)
  • Meta (United States) (US)
Venue
arXiv
Publisher
arXiv
Published
2019
Volume
32
Pages
8548-8559
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1907.05242
arXiv
1907.05242
OpenAlex
W2970401203
Cited by
72
References
0
Citations by year
2025: 3 · 2024: 6 · 2023: 8 · 2022: 5 · 2021: 20 · 2020: 26 · 2019: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
memory-augmented neural networks, structured memory, product key, nearest neighbor search, transformer models, memory layer, inference efficiency, parameter efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review