CoChat
Sign Up
PreprintCited by 49Neural Networks and ApplicationsTopic ModelingAdvanced Neural Network Applications

Block-Recurrent Transformers

Open access
Hutchins, DeLesley, Schlag, Imanol, Wu, Yuhuai, Dyer, Ethan, Neyshabur, Behnam·2022·arXiv
DOI 10.48550/arxiv.2203.07852arXiv 2203.07852
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce the Block-Recurrent Transformer, which applies a transformer layer in a recurrent fashion along a sequence, and has linear complexity with respect to sequence length. Our recurrent cell operates on blocks of tokens rather than single tokens during training, and leverages parallel computation within a block in order to make efficient use of accelerator hardware. The cell itself is strikingly simple. It is merely a transformer layer: it uses self-attention and cross-attention to efficiently compute a recurrent function over a large set of state vectors and tokens. Our design was inspired in part by LSTM cells, and it uses LSTM-style gates, but it scales the typical LSTM cell up by several orders of magnitude. Our implementation of recurrence has the same cost in both computation time and parameter count as a conventional transformer layer, but offers dramatically improved perplexity in language modeling tasks over very long sequences. Our model out-performs a long-range Transformer XL baseline by a wide margin, while running twice as fast. We demonstrate its effectiveness on PG19 (books), arXiv papers, and GitHub source code. Our code has been released as open source.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2203.07852

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 20

  1. [4][4] Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler, 'Long range arena : A benchmark for efficient transformers,' in International Conference on Learning Representations , 2021. -
  2. [6][6] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontañón, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed, 'Big bird: Transformers for longer sequences,' in NeurIPS , 2020. -
  3. [8][8] N. Kitaev, L. Kaiser, and A. Levskaya, 'Reformer: The efficient transformer,' in International Conference on Learning Representations , 2020. -
  4. [10][10] S. Sukhbaatar, D. Ju, S. Poff, S. Roller, A. Szlam, J. Weston, and A. Fan, 'Not all memories are created equal: Learning to forget by expiring,' in ICML , 2021. -
  5. [11][11] Y. Wu, M. Rabe, D. Hutchins, and C. Szegedy, 'Memorizing transformers,' in ICLR , 2022. -
  6. [13][13] H. Ren, H. Dai, Z. Dai, M. Yang, J. Leskovec, D. Schuurmans, and B. Dai, 'Combiner: Full attention transformer with sparse computation cost,' in Advances in Neural Information Processing Systems (A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, eds.), 2021. -
  7. [15][15] J. W. Rae, A. Potapenko, S. M. Jayakumar, C. Hillier, and T. P. Lillicrap, 'Compressive transformers for long-range sequence modelling,' in ICLR , 2020. -
  8. [16][16] Z. Dai, G. Lai, Y. Yang, and Q. Le, 'Funnel-transformer: Filtering out sequential redundancy for efficient language processing,' in NeurIPS , 2020. -
  9. [17][17] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, 'Transformers are RNNs: Fast autoregressive transformers with linear attention,' in ICML , 2020. -
  10. [19][19] H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. A. Smith, and L. Kong, 'Random feature attention,' in ICLR , 2021. -
  11. [20][20] I. Schlag, K. Irie, and J. Schmidhuber, 'Linear Transformers are secretly fast weight programmers,' in ICML , 2021. -
  12. [28][28] V. J. Hellendoorn, P. Maniatis, R. Singh, C. Sutton, and D. Bieber, 'Global relational models of source code,' in ICLR , 2020. -
  13. [36][36] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' in Advances in neural information processing systems , 2017. -
  14. [37][37] R. Csordás, K. Irie, and J. Schmidhuber, 'The neural data router: Adaptive control flow in transformers improves systematic generalization,' in International Conference on Learning Representations , 2022. -
  15. [39][39] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, 'Exploring the limits of transfer learning with a unified text-to-text transformer,' Journal of Machine Learning Research , 2020. -
  16. [41][41] R. K. Srivastava, K. Greff, and J. Schmidhuber, 'Training very deep networks,' in NIPS (C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, eds.), 2015. -
  17. [46][46] Y. Dong, J. Cordonnier, and A. Loukas, 'Attention is not all you need: pure attention loses rank doubly exponentially with depth,' in ICML (M. Meila and T. Zhang, eds.), 2021. -
  18. [49][49] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, 'Language models are few-shot learners,' in NeurIPS , 2020. -
  19. [51][51] N. Shazeer and M. Stern, 'Adafactor: Adaptive learning rates with sublinear memory cost,' in ICML , 2018. -
  20. [53][53] O. Press, N. Smith, and M. Lewis, 'Train short, test long: Attention with linear biases enables input length extrapolation,' in International Conference on Learning Representations , 2022. -

Cited by

Sorted by citations

Related

Authors
Hutchins, D., Schlag, I., Wu, Y., Dyer, E., & Neyshabur, B.
ORCID
Dyer, Ethan
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
33248-33261
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2203.07852
arXiv
2203.07852
OpenAlex
W7133228463
Cited by
43
References
0
Citations by year
2026: 4 · 2025: 11 · 2024: 9 · 2023: 17 · 2022: 2
FWCI
5.39
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
recurrent neural networks, transformer models, sequence modeling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review