CoChat
Sign Up
PreprintCited by 19Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Scaling Transformer to 1M tokens and beyond with RMT

Open access
Bulatov, Aydar, Kuratov, Yuri, Kapushev, Yermek, Burtsev, Mikhail S.·2023·arXiv
DOI 10.48550/arxiv.2304.11062arXiv 2304.11062
Open this paper in CoChat — free accountOpen PDF

Abstract

A major limitation for the broader scope of problems solvable by transformers is the quadratic scaling of computational complexity with input size. In this study, we investigate the recurrent memory augmentation of pre-trained transformer models to extend input context length while linearly scaling compute. Our approach demonstrates the capability to store information in memory for sequences of up to an unprecedented two million tokens while maintaining high retrieval accuracy. Experiments with language modeling tasks show perplexity improvement as the number of processed input segments increases. These results underscore the effectiveness of our method, which has significant potential to enhance long-term dependency handling in natural language understanding and generation tasks, as well as enable large-scale context processing for memory-intensive applications.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.11062

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Token caps in models like GPT-3.5 and GPT-4 pose a significant constraint, particularly in multi-turn conversations.

    “The token caps in models like GPT-3.5 and GPT-4, while extendable through engineering prompts or technologies like Recursive Model Training [4], still pose a significant constraint.”

    From ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning (2024) · cites this paper as [4]

    The Figure 5 passage (Usage of Parameter-Efficient Methods; page unavailable) reports a 32K-token input size for GPT-4 and shows that recurrent memory extended a pre-trained BERT model to much longer sequences. It does not establish a cap for GPT-3.5, test prompt-based extensions, or specifically assess constraints in multi-turn conversations. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 53

  1. Ainslie, J.; Lei, T.; de Jong, M.; Ontañón, S.; Brahma, S.; Zemlyanskiy, Y.; Uthus, D.; Guo, M.; Lee-Thorp, J.; Tay, Y.; Sung, Y.-H.; and Sanghai, S. 2023. CoLT5: Faster Long-Range Transformers with Conditional Computation. arXiv:2303.09752. · arXiv
  2. Ainslie, J.; Ontanon, S.; Alberti, C.; Pham, P.; Ravula, A.; and Sanghai, S. 2020. ETC: Encoding Long and Structured Data in Transformers. arXiv:2004.08483. · arXiv
  3. Peters, M. E.; and Cohan, A. 2020. Long- former: The long-document transformer. arXiv preprint arXiv:2004.05150. · arXiv
  4. Bertsch, A.; Alon, U.; Neubig, G.; and Gormley, M. R. 2023. Unlimiformer: Long-Range Transformers with Un- limited Length Input. arXiv preprint arXiv:2305.01625. · arXiv
  5. Biderman, S.; Schoelkopf, H.; Anthony, Q.; Bradley, H.; O'Brien, K.; Hallahan, E.; Khan, M. A.; Purohit, S.; Prashanth, U. S.; Raff, E.; Skowron, A.; Sutawika, L.; and van der Wal, O. 2023. Pythia: A Suite for Analyz- ing Large Language Models Across Training and Scaling. arXiv:2304.01373. · arXiv
  6. Bulatov, A.; Kuratov, Y.; and Burtsev, M. 2022. Recurrent Memory Transformer. In Koyejo, S.; Mohamed, S.; Agar- wal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., Advances in Neural Information Processing Systems, volume 35, 11079- 11091.
  7. Burtsev, M. S.; Kuratov, Y.; Peganov, A.; and Sapunov, G. V. 2020. Memory transformer. arXiv preprint arXiv:2006.11527. · arXiv
  8. Cho, K.; van Merriënboer, B.; Bahdanau, D.; and Bengio, Y. 2014. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Sta- tistical Translation, 103-111. Doha, Qatar: Association for Computational Linguistics.
  9. Dai, Z.; Yang, Z.; Yang, Y.; Carbonell, J.; Le, Q.; and Salakhutdinov, R. 2019. Transformer-XL: Attentive Lan- guage Models beyond a Fixed-Length Context. In Proceed- ings of the 57th Annual Meeting of the Association for Com- putational Linguistics, 2978-2988. Florence, Italy: Associ- ation for Computational Linguistics.
  10. de Moura, L.; Kong, S.; Avigad, J.; Van Doorn, F.; and von Raumer, J. 2015. The Lean theorem prover (system de- scription). In Automated Deduction-CADE-25: 25th Inter- national Conference on Automated Deduction, Berlin, Ger- many, August 1-7, 2015, Proceedings 25, 378-388. Springer.
  11. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), 4171-4186.
  12. Ding, J.; Ma, S.; Dong, L.; Zhang, X.; Huang, S.; Wang, W.; and Wei, F. 2023. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486. · arXiv
  13. Ding, S.; Shang, J.; Wang, S.; Sun, Y.; Tian, H.; Wu, H.; and Wang, H. 2021. ERNIE-Doc: A Retrospective Long- Document Modeling Transformer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2914-2927. Online: Association for Computational Linguis- tics.
  14. Fan, A.; Lavril, T.; Grave, E.; Joulin, A.; and Sukhbaatar, S. 2020. Addressing some limitations of transformers with feedback memory. arXiv preprint arXiv:2002.09402. · arXiv
  15. Gao, L.; Biderman, S.; Black, S.; Golding, L.; Hoppe, T.; Foster, C.; Phang, J.; He, H.; Thite, A.; Nabeshima, N.; Presser, S.; and Leahy, C. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint arXiv:2101.00027. · arXiv
  16. Graves, A.; Wayne, G.; and Danihelka, I. 2014. Neural tur- ing machines. arXiv preprint arXiv:1410.5401. · arXiv
  17. Graves, A.; Wayne, G.; Reynolds, M.; Harley, T.; Danihelka, I.; Grabska-Barwińska, A.; Colmenarejo, S. G.; Grefen- stette, E.; Ramalho, T.; Agapiou, J.; Badia, A. P.; Hermann, K. M.; Zwols, Y.; Ostrovski, G.; Cain, A.; King, H.; Sum- merfield, C.; Blunsom, P.; Kavukcuoglu, K.; and Hassabis, D. 2016. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626): 471-476.
  18. Grefenstette, E.; Hermann, K. M.; Suleyman, M.; and Blun- som, P. 2015. Learning to Transduce with Unbounded Mem- ory. arXiv:1506.02516. · arXiv
  19. Gu, A.; Goel, K.; and Re, C. 2021. Efficiently Modeling Long Sequences with Structured State Spaces. In Interna- tional Conference on Learning Representations.
  20. Gulcehre, C.; Chandar, S.; and Bengio, Y. 2017. Mem- ory augmented neural networks with wormhole connections. arXiv preprint arXiv:1701.08718. · arXiv
  21. Gulcehre, C.; Chandar, S.; Cho, K.; and Bengio, Y. 2016. Dynamic neural turing machine with soft and hard address- ing schemes. arXiv preprint arXiv:1607.00036. · arXiv
  22. Guo, M.; Ainslie, J.; Uthus, D.; Ontanon, S.; Ni, J.; Sung, Y.-H.; and Yang, Y. 2022. LongT5: Efficient Text-To-Text Transformer for Long Sequences. In Findings of the Asso- ciation for Computational Linguistics: NAACL 2022, 724- 736. Seattle, United States: Association for Computational Linguistics.
  23. Guo, Q.; Qiu, X.; Liu, P.; Shao, Y.; Xue, X.; and Zhang, Z. 2019. Star-Transformer. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 1315-1325. Minneapo- lis, Minnesota: Association for Computational Linguistics.
  24. Gupta, A.; and Berant, J. 2020. GMAT: Global mem- ory augmentation for transformers. arXiv preprint arXiv:2006.03274. · arXiv
  25. He, J.; Zhou, C.; Ma, X.; Berg-Kirkpatrick, T.; and Neubig, G. 2022. Towards a Unified View of Parameter-Efficient Transfer Learning. In International Conference on Learning Representations.
  26. Hochreiter, S.; and Schmidhuber, J. 1997. Long Short-Term Memory. Neural Comput., 9(8): 1735-1780.
  27. Hoffmann, J.; Borgeaud, S.; Mensch, A.; Buchatskaya, E.; Cai, T.; Rutherford, E.; de Las Casas, D.; Hendricks, L. A.; Welbl, J.; Clark, A.; Hennigan, T.; Noland, E.; Millican, K.; van den Driessche, G.; Damoc, B.; Guy, A.; Osindero, S.; Simonyan, K.; Elsen, E.; Vinyals, O.; Rae, J.; and Sifre, L. 2022. An empirical analysis of compute-optimal large lan- guage model training. In Koyejo, S.; Mohamed, S.; Agar- wal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., Advances in Neural Information Processing Systems, volume 35, 30016- 30030.
  28. Hu, E. J.; yelong shen; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations.
  29. Hutchins, D.; Schlag, I.; Wu, Y.; Dyer, E.; and Neyshabur, B. 2022. Block-Recurrent Transformers. In Oh, A. H.; Agar- wal, A.; Belgrave, D.; and Cho, K., eds., Advances in Neural Information Processing Systems.
  30. Joulin, A.; and Mikolov, T. 2015. Inferring Algo- rithmic Patterns with Stack-Augmented Recurrent Nets. arXiv:1503.01007. · arXiv
  31. Lei, J.; Wang, L.; Shen, Y.; Yu, D.; Berg, T. L.; and Bansal, M. 2020. MART: Memory-Augmented Recur- rent Transformer for Coherent Video Paragraph Captioning. arXiv:2005.05402. · arXiv
  32. Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P. 2023. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. · arXiv
  33. Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight De- cay Regularization. In International Conference on Learn- ing Representations.
  34. mathlib Community, T. 2020. The lean mathematical li- brary. In Proceedings of the 9th ACM SIGPLAN Interna- tional Conference on Certified Programs and Proofs. ACM.
  35. McCulloch, W. S.; and Pitts, W. 1943. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5(4): 115-133.
  36. Meng, Y.; and Rumshisky, A. 2018. Context-aware neural model for temporal information extraction. In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), 527-536.
  37. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774. · arXiv
  38. Pang, R. Y.; Parrish, A.; Joshi, N.; Nangia, N.; Phang, J.; Chen, A.; Padmakumar, V.; Ma, J.; Thompson, J.; He, H.; and Bowman, S. 2022. QuALITY: Question Answering with Long Input Texts, Yes! In Proceedings of the 2022 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 5336-5358. Seattle, United States: Association for Compu- tational Linguistics.
  39. Peng, B.; Alcaide, E.; Anthony, Q.; Albalak, A.; Arcadinho, S.; Cao, H.; Cheng, X.; Chung, M.; Grella, M.; GV, K. K.; et al. 2023. RWKV: Reinventing RNNs for the Transformer Era. arXiv preprint arXiv:2305.13048. · arXiv
  40. Krause, A.; Brunskill, E.; Cho, K.; Engelhardt, B.; Sabato, S.; and Scarlett, J., eds., Proceedings of the 40th Interna- tional Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, 28043-28078.
  41. PMLR.
  42. Rae, J. W.; Hunt, J. J.; Harley, T.; Danihelka, I.; Senior, A.; Wayne, G.; Graves, A.; and Lillicrap, T. P. 2016. Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes. arXiv:1610.09027. · arXiv
  43. Rae, J. W.; Potapenko, A.; Jayakumar, S. M.; Hillier, C.; and Lillicrap, T. P. 2020. Compressive Transformers for Long- Range Sequence Modelling. In International Conference on Learning Representations.
  44. Sukhbaatar, S.; Szlam, A.; Weston, J.; and Fergus, R. 2015. End-To-End Memory Networks. arXiv:1503.08895. · arXiv
  45. Sun, Y.; Dong, L.; Huang, S.; Ma, S.; Xia, Y.; Xue, J.; Wang, J.; and Wei, F. 2023. Retentive network: A succes- sor to transformer for large language models. arXiv preprint arXiv:2307.08621. · arXiv
  46. Werbos, P. J. 1990. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10): 1550-1560.
  47. Weston, J.; Bordes, A.; Chopra, S.; and Mikolov, T. 2016. Towards AI-Complete Question Answering: A Set of Pre- requisite Toy Tasks. In Bengio, Y.; and LeCun, Y., eds., 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Confer- ence Track Proceedings.
  48. Weston, J.; Chopra, S.; and Bordes, A. 2015. Memory Net- works. In Bengio, Y.; and LeCun, Y., eds., 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceed- ings.
  49. Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; et al. 2020. Transformers: State-of-the-art natural language pro- cessing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstra- tions, 38-45.
  50. Wu, Q.; Lan, Z.; Qian, K.; Gu, J.; Geramifard, A.; and Yu, Z. 2022a. Memformer: A Memory-Augmented Transformer for Sequence Modeling. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, 308-318. Online only: Association for Computational Linguistics.
  51. Wu, Y.; Rabe, M. N.; Hutchins, D.; and Szegedy, C. 2022b. Memorizing Transformers. In International Conference on Learning Representations.
  52. Zaheer, M.; Guruganesh, G.; Dubey, K. A.; Ainslie, J.; Al- berti, C.; Ontanon, S.; Pham, P.; Ravula, A.; Wang, Q.; Yang, L.; and Ahmed, A. 2020. Big Bird: Transformers for Longer Sequences. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Bal- can, M.; and Lin, H., eds., Advances in Neural Information Processing Systems, volume 33, 17283-17297.
  53. Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; Mihaylov, T.; Ott, M.; Shleifer, S.; Shuster, K.; Simig, D.; Koura, P. S.; Sridhar, A.; Wang, T.; and Zettlemoyer, L. 2022. OPT: Open Pre-trained Transformer Language Models. ArXiv, abs/2205.01068. · arXiv

Cited by

Sorted by citations

Authors
Bulatov, A., Kuratov, Y., Kapushev, Y., & Burtsev, M. S.
Affiliations
  • Moscow Institute of Physics and Technology (RU)
  • AIRI - Artificial Intelligence Research Institute (RU)
  • London Institute for Mathematical Sciences (GB)
ORCID
Kuratov, Yuri
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2304.11062
arXiv
2304.11062
OpenAlex
W4366850543
Cited by
19
References
0
Citations by year
2025: 4 · 2024: 6 · 2023: 9
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Analytical Center for the Government of the Russian Federation (000000D730321P5Q0002, 70-2021-00138)
  • Moscow Institute of Physics and Technology (70-2021-00138)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
transformer models, memory-augmented transformer, long-range dependencies, language modeling, perplexity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review