The NLP Task Effectiveness of Long-Range Transformers
Open accessAbstract
Transformer models cannot easily scale to long sequences due to their O(N 2 ) time and space complexity.This has led to Transformer variants seeking to lower computational complexity, such as Longformer and Performer.While such models have theoretically greater efficiency, their effectiveness on real NLP tasks has not been well studied.We benchmark 7 variants of Transformer models on 5 difficult NLP tasks and 7 datasets.We design experiments to isolate the effect of pretraining and hyperparameter settings, to focus on their capacity for long-range attention.Moreover, we present various methods to investigate attention behaviors to illuminate model details beyond metric scores.We find that the modified attention in long-range transformers has advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This uneven attention compounds approximation errors in longer contexts, such as those necessitated by 300-item IPIP-NEO used here.
“This uneven attention compounds approximation errors in longer contexts53, such as those necessitated by 300-item IPIP-NEO used here, which further motivated our design decision to administer items to models independently.”
From A psychometric framework for evaluating and shaping personality traits in large language models (2025) · cites this paper as [53]
Section 7 reports that XLNet attends less to distant information, while Section 5.4 reports accumulated approximation errors as more Performer layers are stacked. The paper does not say that uneven attention compounds those errors in longer contexts; its error finding concerns transformer depth, not context length. It does not discuss the IPIP-NEO. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827 · see citing works
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · see citing works
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867 · see citing works
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020) · arXiv · doi:10.48550/arxiv.2006.16236 · Cited by 827 · see citing works
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · see citing works
Compressive Transformers for Long-Range Sequence Modelling
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205 · see citing works
Efficient Attentions for Long Document Summarization
Huang, L., Cao, S., Parulian, N., Ji, H., & Wang, L. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.112 · Cited by 146 · see citing works
Coreference Resolution without Span Representations
Kirstain, Y., Ram, O., & Levy, O. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.18653/v1/2021.acl-short.3 · Cited by 60 · see citing works
Do Long-Range Language Models Actually Use Long-Range Context?
Sun, S., Krishna, K., Mattarella-Micke, A., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.62 · Cited by 43 · see citing works
Flagged first, then by citations
Research works printed but not linked: 26
- Artashes Arutiunian, Morgan McGuire, Hallvar Gis- nås, Sheik Mohamed Imran, Dean Pleban, Priyank Negi, and David Arnoldo Ortiz Lozano. 2020. Re- producibility Challenge: Reformer. In Advances in Neural Information Processing Systems (NeurIPS), page 10.
- Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sar- los, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2021. Rethinking Attention with Per- formers. In International Conference on Learning Representations (ICLR). Curation. 2020. Curation Corpus Base.
- Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive Language Models Be- yond a Fixed-Length Context. In Association for Computational Linguistics (ACL).
- Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A Dataset of Information-Seeking Questions and Answers An- chored in Research Papers. In North American Asso- ciation for Computational Linguistics (NAACL).
- Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Pe- ters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A Deep Semantic Natural Language Pro- cessing Platform. In Association for Computational Linguistics (ACL).
- Gautier Izacard and Edouard Grave. 2021. Leverag- ing Passage Retrieval with Generative Models for Open Domain Question Answering. In European Association for Computational Linguistics (EACL).
- Mandar Joshi, Omer Levy, Daniel S. Weld, and Luke Zettlemoyer. 2019. BERT for Coreference Resolu- tion: Baselines and Analysis. In Empirical Methods in Natural Language Processing (EMNLP).
- Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The Efficient Transformer. In In- ternational Conference on Learning Representations (ICLR).
- Yi-An Lai, Garima Lalwani, and Yi Zhang. 2020. Con- text Analysis for Pre-trained Masked Language Mod- els. In Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 3789-3804. Association for Computational Linguistics.
- Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order Coreference Resolution with Coarse-to- fine Inference. In North American Association for Computational Linguistics (NAACL).
- Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Juraf- sky. 2017. Visualizing and Understanding Neural Models in NLP. In Association for Computational Linguistics (ACL).
- Xiaoqiang Luo. 2005. On Coreference Resolution Per- formance Metrics. In Empirical Methods in Natural Language Processing (EMNLP), page 8.
- Richard Yuanzhe Pang, A. Parrish, Nitish Joshi, N. Nan- gia, J. Phang, A. Chen, V. Padmakumar, J. Ma, J. Thompson, H. He, and S. R. Bowman. 2022. QuALITY: Question Answering with Long Input Texts, Yes! In North American Association for Com- putational Linguistics (NAACL).
- Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. 2021b. Random Feature Attention. In International Conference on Learning Representations (ICLR).
- Jack W. Rae and Ali Razavi. 2020. Do Transformers Need Deep Long-Range Memory. In Association for Computational Linguistics (ACL).
- P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD: 100,000+ Questions for Machine Compre- hension of Text. In Empirical Methods in Natural Language Processing (EMNLP).
- Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2020. Efficient Content-Based Sparse Attention with Routing Transformers. Trans- actions of the Association for Computational Linguis- tics (TACL).
- Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020a. Sparse Sinkhorn Attention. In International Conference on Machine Learning (ICML).
- Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020c. Efficient Transformers: A Survey. James Thorne, Majid Yazdani, Marzieh Saeidi, Fabrizio Silvestri, Sebastian Riedel, and Alon Halevy. 2021. Database Reasoning Over Text. In Association for Computational Linguistics (ACL).
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems (NeurIPS).
- Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model- theoretic coreference scoring scheme. In Conference on Message Understanding, page 45. Association for Computational Linguistics.
- R. Weischedel, E. Hovy, M. Marcus, M. Palmer, R. Belvin, S. Pradhan, L. Ramshaw, and N. Xue. 2011. OntoNotes: A large training corpus for en- hanced processing. In Handbook of Natural Lan- guage Processing and Machine Translation. Springer. Springer.
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Empirical Methods in Natural Language Processing (EMNLP), pages 38-45. Association for Computa- tional Linguistics.
- Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhut- dinov, and Q. V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understand- ing. In Advances in Neural Information Processing Systems (NeurIPS).
- Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021. DocNLI: A Large-scale Dataset for Document- level Natural Language Inference. In Association for Computational Linguistics (ACL).
- Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big Bird: Transformers for Longer Sequences. In Advances in Neural Informa- tion Processing Systems (NeurIPS).
Cited by
Sorted by citations
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · see citing works
“Qin et al. (2023) analyze how efficient Transformers perform on a variety of long-context downstream NLP tasks, finding that long-context transformers are recency-biased and do not effectively use long-range context.”
BackgroundA psychometric framework for evaluating and shaping personality traits in large language models
Serapio-García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Romero, P., Abdulhai, M., Faust, A., & Matarić, M. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01115-6 · Cited by 30 · see citing works
MARG: Multi-Agent Review Generation for Scientific Papers
D'Arcy, M., Hope, T., Birnbaum, L., & Downey, D. (2024) · arXiv · doi:10.48550/arxiv.2401.04259 · Cited by 9 · see citing works
- Authors
- Qin, G., Feng, Y., & Van Durme, B.
- Affiliations
- Johns Hopkins University (US)
- Venue
- Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
- Published
- 2023
- Pages
- 3774-3790
- OpenAlex
- W4386576812
- Cited by
- 14
- References
- 9
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 5 · 2023: 3
- FWCI
- 2.48
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the European Chapter of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Intelligence Advanced Research Projects Activity
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- long-range attention, Longformer, performers, computational complexity, attention mechanism, pre-training, hyperparameter settings, content selection, approximation error
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
