Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Open accessAbstract
Since the introduction of the transformer model by Vaswani et al. (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training? We first show that extrapolation can be enabled by simply changing the position representation method, though we find that current methods do not allow for efficient extrapolation. We therefore introduce a simpler and more efficient position method, Attention with Linear Biases (ALiBi). ALiBi does not add positional embeddings to word embeddings; instead, it biases query-key attention scores with a penalty that is proportional to their distance. We show that this method trains a 1.3 billion parameter model on input sequences of length 1024 that extrapolates to input sequences of length 2048, achieving the same perplexity as a sinusoidal position embedding model trained on inputs of length 2048 but training 11% faster and using 11% less memory. ALiBi's inductive bias towards recency also leads it to outperform multiple strong position methods on the WikiText-103 benchmark.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The additive pre-softmax form is compatible with FlashAttention.
“The additive pre-softmax form is compatible with FlashAttention [2,13,23] .”
From Inference-Time Attention Steering for Vision-Language-Action Driving Models (2026) · cites this paper as [13]
The Figure 3 caption (p. 5) says ALiBi adds a bias to attention scores before applying softmax, supporting the description of an additive pre-softmax form. The paper does not address compatibility with FlashAttention, so it does not support that part of the claim. (LLM, conf=0.86)
Experimental — accuracy not yet measuredUnlimited context remains computationally infeasible and doesn't address selective retention or consistency.
“unlimited context remains computationally infeasible and doesn't address selective retention or consistency.”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [23]
The Introduction (p. 1) reports that a longer-input T5 run exhausted a 32GB GPU, and Appendix B.2 (p. 24) calls sliding-window inference very slow. These support a computational constraint on longer contexts, but not the categorical claim about unlimited context. The paper does not address selective retention or consistency. (LLM, conf=0.78)
Experimental — accuracy not yet measuredRecent works extend context windows to 100K+ tokens.
“Recent works extend context windows to 100K+ tokens [18], [19], [23].”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [23]
The Introduction (p. 2) says ALiBi enables inference on longer sequences, and §4.2 (p. 8) reports strong performance up to 10,000 tokens. These passages support context-window extension, but not the attributed magnitude of 100K+ tokens. (LLM, conf=0.88)
Experimental — accuracy not yet measuredLocal-window memory approaches (Alibi position encoding, sliding-window attention) also struggle on the multi-unrelated-documents question-answering task for the same reason.
“Similarly, local-window memory approaches, such as Alibi position encoding Press et al. [2021] and sliding-window attention Beltagy et al. [2020], also struggle for the same reason.”
From Memory Mosaics at scale (2025) · cites this paper as Press et al., 2021
The Introduction (p. 1) identifies ALiBi as a method that penalizes attention to more distant tokens. Appendix B.2 (p. 24) suggests that ALiBi might not use contexts longer than those seen in training. The paper does not evaluate multi-unrelated-documents question answering or establish that ALiBi struggles on that task for the reason claimed. (LLM, conf=0.82)
Experimental — accuracy not yet measuredModels employing relative position approaches may exhibit improved performance in extrapolating to longer sequences compared to those utilizing absolute position methods.
“Studies suggest that models employing relative position approaches may exhibit improved performance in extrapolating to longer sequences compared to those utilizing absolute position methods (Press et al. 2021).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Press et al. 2021
Positional embeddings greatly affect Transformers' ability to generalize to longer lengths.
“Positional embeddings have been shown to greatly affect Transformers' ability to generalize to longer lengths (Press et al., 2021;Kazemnejad et al., 2024;Ruoss et al., 2023;Su et al., 2024;Li et al., 2023;Cho et al., 2024;Sabbaghi et al., 2024;McLeish et al., 2024;Golovneva et al., 2024).”
From Looped Transformers for Length Generalization (2024) · cites this paper as Press et al., 2021
Sparse-attention and positional bias studies prove that models struggle to maintain strict instruction salience across multi-turn conversational updates.
“sparse-attention (Longformer, BigBird) [10, 11] and positional bias studies (ALiBi) [12] prove that while models can physically process long sequences, they struggle to maintain strict instruction salience across multi-turn conversational updates.”
From HierMem: Context Curation Over Context Scaling — Hierarchical Memory with Invariant Constraint Placement for Long-Horizon LLM Conversations (2026) · cites this paper as [12]
The Introduction (p. 1) reports how positional methods affect perplexity as input sequences lengthen; it does not test whether instructions remain salient. Searches for instruction salience and multi-turn conversation found no passage addressing conversational updates, so this paper does not establish the attributed conclusion. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 30 · Cited by 161
Research works printed but not linked: 26
- Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. CoRR , abs/1809.10853, 2018. URL http://arxiv.org/abs/1809.10853 . · arXiv
- Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150 , 2020. · arXiv
- Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm´ an, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 2020. doi: 10.18653/v1/2020.acl-main.747. URL http://dx.doi.org/10.18653/v1/2020.acl-main.747 . · DOI · Paper page
- Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1285. URL https://aclanthology.org/P19-1285 . · DOI · Paper page
- Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam M. Shazeer, Andrew M. Dai, M. Hoffman, M. Dinculescu, and D. Eck. Music transformer: Generating music with long-term structure. In ICLR , 2019.
- Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research , 67:757-795, April 2020. doi: 10.1613/jair.1.11674. URL https://doi.org/10.1613/jair.1. 11674 . · DOI · Paper page
- Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In ICLR , 2017. URL https://openreview.net/ forum?id=r1aPbsFle .
- Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through Memorization: Nearest Neighbor Language Models. In International Conference on Learning Representations (ICLR) , 2020.
- Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. Shape: Shifted absolute position embedding for transformers. ArXiv , abs/2109.05644, 2021. · arXiv
- Andrew Kyle Lampinen, Stephanie C. Y. Chan, Andrea Banino, and Felix Hill. Towards mental time travel: a hierarchical memory for reinforcement learning agents. CoRR , abs/2105.14039, 2021. URL https://arxiv.org/abs/2105.14039 . · arXiv
- Tatiana Likhomanenko, Qiantong Xu, Ronan Collobert, Gabriel Synnaeve, and Alex Rogozhnikov. CAPE: encoding relative positions with continuous augmented positional embeddings. CoRR , abs/2106.03143, 2021. URL https://arxiv.org/abs/2106.03143 . · arXiv
- Tomas Mikolov and G. Zweig. Context dependent recurrent neural network language model. 2012 IEEE Spoken Language Technology Workshop (SLT) , pp. 234-239, 2012.
- Masato Neishi and Naoki Yoshinaga. On the relation between position information and sentence length in neural machine translation. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) , pp. 328-338, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/K19-1031. URL https://aclanthology.org/K19-1031 . · DOI
- Benjamin Newman, John Hewitt, Percy Liang, and Christopher D. Manning. The eos decision and length extrapolation. In BlackBoxNLP@EMNLP , 2020. URL https://nlp.stanford. edu/pubs/newman2020extrapolation.pdf .
- Rodrigo Nogueira, Zhiying Jiang, and Jimmy J. Li. Investigating the limitations of the transformers with simple arithmetic tasks. ArXiv , abs/2102.13019, 2021. · arXiv
- Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation (WMT) , 2018.
- Ankur Parikh, Oscar T¨ ackstr¨ om, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pp. 2249-2255, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1244. URL https: //aclanthology.org/D16-1244 . · DOI
- Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers , pp. 157-163, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/E17-2025 .
- Ofir Press, Noah A. Smith, and Mike Lewis. Shortformer: Better language modeling using shorter inputs. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 5493-5505, Online, August 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.acl-long.427 .
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id= SylKikSYDH .
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-totext transformer. Journal of Machine Learning Research , 21(140):1-67, 2020. URL http: //jmlr.org/papers/v21/20-074.html .
- Jan Rosendahl, Viet Anh Khoa Tran, Weiyue Wang, and Hermann Ney. Analysis of positional encodings for neural machine translation. In International Workshop on Spoken Language Translation , Hong Kong, China, November 2019.
- Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pp. 464-468, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-2074. URL https://www.aclweb.org/anthology/N18-2074 . · DOI · Paper page
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R´ emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pp. 38-45, Online, October 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/ 2020.emnlp-demos.6 .
- Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. DA-transformer: Distance-aware transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 2059-2068, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.166. URL https://aclanthology.org/2021.naacl-main.166 . · DOI
- Authors
- Press, O., Smith, N. A., & Lewis, M.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2108.12409
- OpenAlex
- W3196318247
- Cited by
- 161
- References
- 30
- Citations by year
- 2026: 6 · 2025: 34 · 2024: 50 · 2023: 47 · 2022: 22 · 2021: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- position representation, positional encoding, inductive bias, long sequence modeling, transformer models, training efficiency, memory efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
