Efficient Long-Text Understanding with Short-Text Models
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Transformer-based pretrained language models (LMs) are ubiquitous across natural language understanding, but cannot be applied to long sequences such as stories, scientific articles, and long documents due to their quadratic complexity. While a myriad of efficient transformer variants have been proposed, they are typically based on custom implementations that require expensive pretraining from scratch. In this work, we propose SLED: SLiding-Encoder and Decoder, a simple approach for processing long sequences that re-uses and leverages battle-tested short-text pretrained LMs. Specifically, we partition the input into overlapping chunks, encode each with a short-text LM encoder and use the pretrained decoder to fuse information across chunks (fusion-in-decoder). We illustrate through controlled experiments that SLED offers a viable strategy for long text understanding and evaluate our approach on SCROLLS, a benchmark with seven datasets across a wide range of language understanding tasks. We find that SLED is competitive with specialized models that are up to 50x larger and require a dedicated and expensive pretraining step.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our experimental setup is similar to the needle-in-a-haystack experiments of Ivgi et al. (2023), who compare question answering performance when the relevant paragraph is placed (i) at the beginning of the input or (ii) a random position within the input.They find that encoder-decoder models have significantly higher performance when relevant information is placed at the start of the input context.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Ivgi et al. (2023)
In the citing paper
“They find that encoder-decoder models have significantly higher performance when relevant information is placed at the start of the input context.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Ivgi et al. (2023)
In the citing paper
“This category is a real-world adaptation (Ivgi et al., 2023) of NIAH”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Ivgi et al., 2023
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“Recent advances in natural language processing have been by and large fueled by the transformer architecture (Vaswani et al., 2017).”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“Our approach can be readily applied to any pretrained encoder-decoder LM such as T5 (Raffel et al., 2020b) and BART (Lewis et al., 2020) (but is not applicable to decoder-only (Brown et al., 2020) or encoder-only models (Liu et al., 2019; Conneau et al., 2020)).”
MethodAdvances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 5×
“, 2020) or T5 (Raffel et al., 2020b), and embedding it into SLED’s framework results in dramatic improvement in performance (6 points on average across models).”
BackgroundMethodInfluential(2004) · Cited by 9,718
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 2×
“To check if SLED can ignore irrelevant text and locate a single piece of information, we cast SQuAD 1.1 (Rajpurkar et al., 2016) as a sequence-to-sequence task with long input.”
MethodInfluential
Research works printed but not linked: 4
- Raffel, 2020, Exploring the limits of transfer learning with a unified text-to-text transformer, ArXiv
- Yi, 2022, Unifying language learning paradigms, ArXiv
- Xiong, 2022, Simple local attentions remain competitive for long-context tasks, 1975, 10.18653/v1/2022.naacl-main.144 · DOI
- Zaheer, 2020, Big bird: Transformers for longer sequences
Cited by
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
“Our experimental setup is similar to the needle-in-a-haystack experiments of Ivgi et al. (2023), who compare question answering performance when the relevant paragraph is placed (i) at the beginning of the input or (ii) a random position within the input.”
MethodLost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2307.03172 · Cited by 1,599
“Our experimental setup is similar to the needle-in-a-haystack experiments of Ivgi et al. (2023), who compare question answering performance when the relevant paragraph is placed (i) at the beginning of the input or (ii) a random position within the input.”
MethodResource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 276
“SLED [108], short for SLiding-Encoder and Decoder, repurposes and capitalizes on well-validated short-text pre-trained language models.”
An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Meng, X., Zhang, J., Liu, Y., Wang, W., Li, S., & Zhang, Y. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00181 · Cited by 192
“For example, we can adopt a sliding-encoder and decoder (SLED) [26] on the pre-trained models to accept long and/or dependent methods.”
MethodParallel Context Windows for Large Language Models
Ratner, N., Levine, Y., Belinkov, Y., Ram, O., Magar, I., Abend, O., Karpas, E., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.352 · Cited by 111
“Ivgi et al. (2022) suggest SLED, an encoder–decoder model for long texts, which encodes short overlapping chunks of the input text, and fuses the information in the decoder, a-la Fusion-in-Decoder (Izacard and Grave, 2020).”
BackgroundMethodLEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
Niklaus, J., Matoshi, V., Rani, P., Galassi, A., Stürmer, M., & Chalkidis, I. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.200 · Cited by 50
Investigating Efficiently Extending Transformers for Long Input Summarization
Phang, J., Zhao, Y., & Liu, P. X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.240 · Cited by 83
Efficient Memory-Enhanced Transformer for Long-Document Summarization in Low-Resource Regimes
Moro, G., Ragazzi, L., Valgimigli, L., Frisoni, G., Sartori, C., & Marfia, G. (2023) · Sensors · doi:10.3390/s23073542 · Cited by 68
“Segmentation-based approaches [24–26] with fusion-in-decoder [27] and marginalized decoding [28] divide the input into meaningful units to produce a summary.”
MethodIn-Context Learning with Long-Context Models: An In-Depth Exploration
Bertsch, A., Ivgi, M., Xiao, E., Alon, U., Berant, J., Gormley, M. R., & Neubig, G. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.605 · Cited by 177
Self-citationContext Length Alone Hurts LLM Performance Despite Perfect Retrieval
Du, Y., Tian, M., Ronanki, S., Rongali, S., Bodapati, S. B., Galstyan, A., Wells, A., Schwartz, R., Huerta, E. A., & Peng, H. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.1264 · Cited by 39
Retrieval-Augmented Generation with Graphs (GraphRAG)
Han, H., Wang, Y., Shomer, H., Guo, K., Ding, J., Lei, Y., Halappanavar, M., Rossi, R. A., Mukherjee, S., Tang, X., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.00309 · Cited by 34
Comprehensive Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Zhang, J., Bao, X., Wang, X., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3532759 · Cited by 56
“Leveraging high-level semantic information (such as bytecode [117] and intermediate representation [118]) can aid the model in comprehending the root cause of defects, thereby enhancing its repair capability.”
ZeroSCROLLS: A zero-shot benchmark for long text understanding
Shaham, U., Ivgi, M., Efrat, A., Berant, J., & Levy, O. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.536 · Cited by 23
Self-citationRetrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval
Rubin, O., & Berant, J. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00693 · Cited by 44
Self-citationLEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal DomainPreprint
Niklaus, J., Matoshi, V., Rani, P., Galassi, A., Stuermer, M., & Chalkidis, I. (2023) · arXiv (Cornell University) · doi:10.24451/arbor.19688 · Cited by 50
Chakraborty, P., Alfadel, M., & Nagappan, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3579574 · Cited by 29
“It demonstrates enhancements over SemanticCodeBERT, improving by 23% in both Top 1 and MAP, and 18% in MRR. Similarly, against FBL-BERT, BLAZE achieves substantial performance increases of 60% in Top 1, 50% in MAP, and 44% in MRR. Whenanalyzing the performance of BLAZE across the studied programming languages, we find…”
InfluentialRULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · doi:10.48550/arxiv.2404.06654 · Cited by 1,094
“The synthetic input in R ULER reduces reliance on parametric knowledge, which interferes with the utilization of long-context input in realistic tasks (Shaham et al., 2023; Bai et al., 2023).”
BackgroundProceedings of the 4th New Frontiers in Summarization Workshop
(2023) · doi:10.18653/v1/2023.newsum-1 · Cited by 12
MARG: Multi-Agent Review Generation for Scientific Papers
D'Arcy, M., Hope, T., Birnbaum, L., & Downey, D. (2024) · arXiv · doi:10.48550/arxiv.2401.04259 · Cited by 9
NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents
Czinczoll, T., Hönes, C. J., Schall, M., & de Melo, G. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.256 · Cited by 16
“…modeling long-range dependencies via a graph structure (Zhang et al., 2022; Bugueño and de Melo, 2023), as well as compressing the input either via an explicit memory module (Rae et al., 2019; Zemlyanskiy et al., 2021) or via hierarchical processing (Pappagari et al., 2019; Ivgi et al., 2022).”
BackgroundCluster-Aggregated Transformer: Enhancing lightweight parameter models
Guo, Z., Adedigba, A. P., & Mallipeddi, R. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111468 · Cited by 8
End-to-End Long Document Summarization using Gradient Caching
Saxena, R., Tang, H., & Keller, F. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.40 · Cited by 12
Improving Multi-Stage Long Document Summarization with Enhanced Coarse Summarizer
Lim, J., & Song, H.-J. (2023) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2023.newsum-1.13 · Cited by 6
ELICE: Embedding Language through Informative Contrastive-Encoder
Chiao, C.-S., & Lin, D.-T. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120523 · Cited by 4
Enhancing text comprehension for Question Answering with Contrastive Learning
Lee, S., & Lee, M. (2023) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2023.repl4nlp-1.7 · Cited by 4
“- SLED 5 (Ivgi et al., 2023) is a method of processing long texts using a fusion-in-decoder-based encoder-decoder language model.”
MethodTang, J., Dou, W., Shen, D., Nie, T., & Kou, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5569-1_20 · Cited by 4
Beyond decomposition: Hierarchical dependency management in multi‐document question answering
Zheng, X., Li, Z., Chen, Q., & Zhang, Y. (2024) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24971 · Cited by 3
“Handling this task requires language models to operate over long sequences (Dong et al., 2023; Ivgi et al., 2023; Liu et al., 2024; Nair et al., 2023; Xiong et al., 2023).”
Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP
Goldman, O., Jacovi, A., Slobodkin, A., Maimon, A., Dagan, I., & Tsarfaty, R. (2024) · arXiv · doi:10.48550/arxiv.2407.00402 · Cited by 3
“For example, Needle in a Haystack tasks (NIAH; Ivgi et al., 2023; Mohtashami and Jaggi, 2023) involve queries whose main challenge is finding the relevant information in a long context, without requiring much further processing.”
Human-inspired Episodic Memory for Infinite Context LLMs
Fountas, Z., Benfeghoul, M. A., Oomerjee, A., Christopoulou, F., Lampouras, G., Bou-Ammar, H., & Wang, J. (2024) · arXiv · doi:10.48550/arxiv.2407.09450 · Cited by 3
ELCA: Enhanced boundary location for Chinese named entity recognition via contextual association
Wang, Y., Mao, S., & Jiang, Y. (2024) · Intelligent Data Analysis · doi:10.3233/ida-230383 · Cited by 2
LongT5-Mulla: LongT5 With Multi-Level Local Attention for a Longer Sequence
Zhou, L. (2023) · IEEE Access · doi:10.1109/access.2023.3340854 · Cited by 6
“As another research route, DANCER [11], Hi-Transformer [13], SummˆN [12], and SLED [24] design different hierarchical architectures that can similarly block-wise process small parts of input text, and their main difference to recurrent patterns is that they utilize high-level Transformers to…”
BackgroundMethodDon't Use a Cannon to Kill a Fly: An Efficient Cascading Pipeline for Long Documents
Li, Z., Guha, N., & Nyarko, J. (2023) · International Conference on Artificial Intelligence and Law · doi:10.1145/3594536.3595142 · Cited by 2
Improving Long Text Understanding with Knowledge Distilled from Summarization Model
Liu, Y., Yang, Y., & Chen, X. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448039 · Cited by 7
“Transformer-based models are prevalent [13, 14, 15, 16, 17, 18] across numerous NLP tasks[19], but have difficulty in processing long texts due to the quadratic complexity of input text length[20].”
BackgroundA Domain-Adaptive Large Language Model With Refinement Framework For IoT Cybersecurity
Che, X., Yu, Z., Zhu, M., Li, Q., & Xu, D. (2024) · IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) · doi:10.1109/ithings-greencom-cpscom-smartdata-cybermatics62450.2024.00056 · Cited by 4
“Therefore, inspired by [6], we propose a text understanding-based refinement framework to enhance the original corpus.”
BackgroundMotghare, M., & Agrawal, A. (2024) · International Conference on Artificial Intelligence and Quantum Computation-Based Sensor Application (ICAIQSA) · doi:10.1109/icaiqsa64000.2024.10882315 · Cited by 1
“SLED[41] offers an efficient approach to summarizing long documents by employing a sliding encoder-decoder mechanism, which facilitates effective long text processing without extensive retraining.”
Unsupervised Context-Linking Retriever for Question Answering on Long Narrative Books
Ateeq, M. A., Tiun, S. B., Abdelhaq, H., & Kaur, W. (2025) · IEEE Access · doi:10.1109/access.2025.3578497 · Cited by 1
Text-Enhanced Method for LLMs Domain Adaptation in Cybersecurity
Che, X., Gao, P., Zhu, M., Gu, H., Li, Q., & Li, M. (2025) · Journal of Computer Information Systems · doi:10.1080/08874417.2025.2554853 · Cited by 1
Ruan, W., Qi, T., He, J. Y. D., Sun, B., & Zheng, G. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228999 · Cited by 1
“In the case of Chinese language teaching, large language models (LLMs) often struggle with the complexity of generating content that fits these precise requirements due to their quadratic complexity[1].”
The extraction of a brief summary from scientific documents using machine learning methods
Murzabekova, G., Mukhamedrakhimova, G., Taszhurekova, Z., Yerbayev, Y., Доумчариева, Ж., Makhatova, V. E., Tolganbaeva, M., & Serikbayeva, S. (2025) · Bulletin of Electrical Engineering and Informatics · doi:10.11591/eei.v14i6.10660 · Cited by 1
“Extracting summaries from scientific documents [11]-[13] has become an important task that demands the use of modern natural language processing (NLP) methods [14]-[18].”
Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-To-Coarse Attention
He, Z., Yuan, J., Zhou, L., Leng, J., & Jiang, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446483 · Cited by 2
“Onelineof research respects the length limitation by partitioning a long input into smaller segments and feeding them separately to models pretrained on short texts [6, 7].”
BackgroundAdel, A. A. (2023) · Studies in computational intelligence · doi:10.1007/978-3-031-44865-2_10
Focus Your Attention (with Adaptive IIR Filters)
Lutati, S., Zimerman, I., & Wolf, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.772 · Cited by 13
Long Text Classification with Segmentation
Mayatskaya, E. (2024) · Ural Symposium on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT) · doi:10.1109/usbereit61901.2024.10583985 · Cited by 1
“In the method of [10], the authors also use text segmentation into overlapping blocks, where they encode each block using an encoder.”
MethodEntity-enhanced denoising text summarization for social media with sparse attention mechanism
Meng, X., Zhang, J., & Liu, J. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2025.11.051
ScaleFormer: Span Representation Cumulation for Long-Context Transformer
Du, J., Yin, W., & Yu, P. (2025) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3767695.3769513
Reader comes first: A demand-oriented readability controllable summarization
Han, Q., Sun, Y., Yang, Z., Liu, W., Luo, L., & Lin, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130858
KI-unterstützter Filmschnitt – Chancen und Herausforderungen
Döbert, A. (2025) · Springer eBooks · doi:10.1007/978-3-658-47898-8_1-1
A survey of hate speech detection in indian languages
Jerin Mahibha, C., & Thenmozhi, D. (2026) · Language Resources and Evaluation · doi:10.1007/s10579-025-09899-0
“The concise nature of short texts poses a challenge for conventional ML algorithms, which represents the requirement of specialized modeling techniques (Ivgi et al., 2023 ).”
Palanisamy, B., Chalapathi, G., & Hassija, V. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11580-z
Agentic Self-RAG: An Ablation Study for Zero-Shot Biomedical Question Answering
Wibawa, R. P., Putri, S. W., Wijaya, A. L., & Hidayaturrahman (2026) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict71158.2026.11621038
“Fragments are limited to remain within the 512 token limit of the MedCPT embedding model [5], with a two-sentence sliding window to ensure contextual continuity between fragments [28].”
An Intelligent English Grammar Error Correction Method Based on the BERT Pre-trained Language Model
Li, D. (2026) · International Conference on Machine Intelligence and Digital Applications (MIDA) · doi:10.1145/3801438.3806610
“However, these methods typically process text in a unidirectional manner and rely heavily on sequential representations, which limits their ability to capture long-range dependencies and rich contextual semantics [11].”
Enabling Retriever-LLM Connection across the Semantic Gap in Retrieval-Augmented Generation
Ye, F.-D., Li, S.-Y., Zhang, Y.-Q., Chen, L., & Qu, S.-B. (2026) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3847108
“However, these implicit aggregation techniques are typically designed for encoder-decoder LLMs, limiting their suitability to the prevalent encoder-only retrievers and decoder-only LLMs [31].”
Xu, N., & Ma, X. (2024) · arXiv · doi:10.48550/arxiv.2410.14166
Focus Your Attention (with Adaptive IIR Filters)
Lutati, S., Zimerman, I., & Wolf, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14952
“Recent studies have demonstrated that a combination of local and global transformers can achieve state-of-the-art performance on long-range tasks (Ivgi et al., 2023; Yu et al., 2023; Hutchins et al., 2022).”
Background
Metadata
- Authors
- Ivgi, M., Shaham, U., & Berant, J.
- Affiliations
- Tel Aviv University (IL)
- Corresponding author
- Ivgi, Maor
- Shaham, Uri
- Berant, Jonathan
- ORCID
- Shaham, Uri
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2023
- Volume
- 11
- Pages
- 284-299
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4353004366
- Cited by
- 42
- References
- 50
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 11 · 2023: 15
- FWCI
- 3.72
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- European Commission
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- efficient transformers, chunking, pre-trained language models, quadratic complexity, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.