Shortformer: Better Language Modeling using Shorter Inputs
Open accessAbstract
Increasing the input length has been a driver of progress in language modeling with transformers. We identify conditions where shorter inputs are not harmful, and achieve perplexity and efficiency improvements through two new methods that decrease input length. First, we show that initially training a model on short subsequences before moving on to longer ones both reduces overall training time and, surprisingly, substantially improves perplexity. Second, we show how to improve the efficiency of recurrence methods in transformers, which let models condition on previously processed tokens when generating sequences that exceed the maximal length the transformer can handle at once. Existing methods require computationally expensive relative position embeddings; we introduce a simple alternative of adding absolute position embeddings to queries and keys instead of to word embeddings, which efficiently produces superior results. We show that these recurrent models also benefit from short input lengths. Combining these techniques speeds up training by a factor of 1.65, reduces memory usage, and substantially improves perplexity on WikiText-103, without adding any parameters.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Press et al. (2020) observe a phenomenon called the 'Early Token Curse': tokens with a greater amount of context preceding them are fundamentally easier to predict, whereas the first several tokens a model must predict 'from scratch' are difficult or impossible to predict without information to condition on.
“However, Press et al. (2020) observe a phenomenon they call the 'Early Token Curse', referring to the fact that tokens with a greater amount of context preceding them are fundamentally easier to predict, whereas the first several tokens a model must predict 'from scratch' are difficult or impossible to predict without information to condition on.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Press et al. (2020)
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredPress et al. (2020) propose a strided or sliding window perplexity evaluation method to mitigate the early token curse.
“To mitigate this issue, they propose a strided or sliding window perplexity evaluation method.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Press et al. (2020)
The Introduction and §3.1 explain that sliding-window evaluation gives early tokens more context and improves perplexity, addressing the early token curse. But the Introduction attributes the sliding-window method to Baevski and Auli (2018); Press et al. use and analyze it rather than propose it as a new method. (LLM, conf=0.86)
Experimental — accuracy not yet measuredPress et al. (2020) demonstrate how the methodology typically used-'nonoverlapping windows'-to compute perplexity on greater-than-context-length sequences systematically skews results in favor of models trained and thus evaluated with longer context lengths.
“For example, Press et al. (2020) demonstrate how the methodology typically used-'nonoverlapping windows' (Appendix B)-to compute perplexity on greater-thancontext-length sequences systematically skews results in favor of models trained and thus evaluated with longer context lengths.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Press et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 17
- Alexei Baevski and Michael Auli. 2018. Adaptive in- put representations for neural language modeling. CoRR, abs/1809.10853. · arXiv
- Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150. · arXiv
- Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Con- ference on Machine Learning, ICML '09, page 41-48, New York, NY, USA. Association for Com- puting Machinery. · DOI · Paper page
- Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Car- bonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 2978-2988.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Associ- ation for Computational Linguistics. · DOI · Paper page
- Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. Tying word vectors and word classifiers: A loss framework for language modeling. In ICLR.
- Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through Memorization: Nearest Neighbor Language Models. In International Conference on Learning Representations (ICLR).
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Inter- national Conference on Learning Representations.
- Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. · arXiv
- Ofir Press, Noah A. Smith, and Omer Levy. 2020. Im- proving transformer models by reordering their sub- layers. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 2996-3005, Online. Association for Computa- tional Linguistics. · DOI · Paper page
- Ofir Press and Lior Wolf. 2017. Using the output em- bedding to improve language models. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157-163, Valencia, Spain. Association for Computational Linguistics.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayaku- mar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In International Conference on Learn- ing Representations.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former. arXiv preprint arXiv:1910.10683. · arXiv
- Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position represen- tations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 464-468, New Orleans, Louisiana. Association for Computa- tional Linguistics. · DOI · Paper page
- Sainbayar Sukhbaatar, Edouard Grave, Piotr Bo- janowski, and Armand Joulin. 2019. Adaptive at- tention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 331-335, Florence, Italy. Association for Computational Linguistics. · DOI
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, pages 5998-6008.
- Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE inter- national conference on computer vision, pages 19- 27.
Cited by
Sorted by citations
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation
Liu, T., Zhang, Y., Brockett, C., Mao, Y., Sui, Z., Chen, W., & Dolan, B. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.464 · Cited by 63 · see citing works
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9 · see citing works
- Authors
- Press, O., Smith, N. A., & Lewis, M.
- Affiliations
- University of Washington (US)
- University of Pittsburgh (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Pages
- 5493-5505
- arXiv
- 2012.15832
- OpenAlex
- W3118895645
- Cited by
- 55
- References
- 0
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 6 · 2023: 13 · 2022: 14 · 2021: 13 · 2020: 1
- FWCI
- 5.27
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- language modeling, efficient language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
