CoChat

Efficient Long-Text Understanding with Short-Text Models

Open access
Ivgi, Maor, Shaham, Uri, Berant, Jonathan · 2023 · Transactions of the Association for Computational Linguistics

The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.

Abstract

Transformer-based pretrained language models (LMs) are ubiquitous across natural language understanding, but cannot be applied to long sequences such as stories, scientific articles, and long documents due to their quadratic complexity. While a myriad of efficient transformer variants have been proposed, they are typically based on custom implementations that require expensive pretraining from scratch. In this work, we propose SLED: SLiding-Encoder and Decoder, a simple approach for processing long sequences that re-uses and leverages battle-tested short-text pretrained LMs. Specifically, we partition the input into overlapping chunks, encode each with a short-text LM encoder and use the pretrained decoder to fuse information across chunks (fusion-in-decoder). We illustrate through controlled experiments that SLED offers a viable strategy for long text understanding and evaluate our approach on SCROLLS, a benchmark with seven datasets across a wide range of language understanding tasks. We find that SLED is competitive with specialized models that are up to 50x larger and require a dedicated and expensive pretraining step.

Overview

Integrity signals

Registry

Crossref · 10.1162/tacl_a_00547

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 51 references

Research works printed but not linked: 4

  1. Raffel, 2020, Exploring the limits of transfer learning with a unified text-to-text transformer, ArXiv
  2. Yi, 2022, Unifying language learning paradigms, ArXiv
  3. Xiong, 2022, Simple local attentions remain competitive for long-context tasks, 1975, 10.18653/v1/2022.naacl-main.144 · DOI
  4. Zaheer, 2020, Big bird: Transformers for longer sequences

Cited by

  • Lost in the Middle: How Language Models Use Long Contexts

    Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612

    “Our experimental setup is similar to the needle-in-a-haystack experiments of Ivgi et al. (2023), who compare question answering performance when the relevant paragraph is placed (i) at the beginning of the input or (ii) a random position within the input.”
    Method
  • Lost in the Middle: How Language Models Use Long Contexts

    Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2307.03172 · Cited by 1,599

    “Our experimental setup is similar to the needle-in-a-haystack experiments of Ivgi et al. (2023), who compare question answering performance when the relevant paragraph is placed (i) at the beginning of the input or (ii) a random position within the input.”
    Method
  • Resource-efficient Algorithms and Systems of Foundation Models: A Survey

    Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2025) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 277

    “SLED [108], short for SLiding-Encoder and Decoder, repurposes and capitalizes on well-validated short-text pre-trained language models.”
  • An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair

    Huang, K., Meng, X., Zhang, J., Liu, Y., Wang, W., Li, S., & Zhang, Y. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00181 · Cited by 192

    “For example, we can adopt a sliding-encoder and decoder (SLED) [26] on the pre-trained models to accept long and/or dependent methods.”
    Method
  • Parallel Context Windows for Large Language Models

    Ratner, N., Levine, Y., Belinkov, Y., Ram, O., Magar, I., Abend, O., Karpas, E., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.352 · Cited by 111

    “Ivgi et al. (2022) suggest SLED, an encoder–decoder model for long texts, which encodes short overlapping chunks of the input text, and fuses the information in the decoder, a-la Fusion-in-Decoder (Izacard and Grave, 2020).”
    BackgroundMethod

See all 54

Metadata

Authors
Ivgi, M., Shaham, U., & Berant, J.
Affiliations
Tel Aviv University (IL)
Corresponding author
  • Ivgi, Maor
  • Shaham, Uri
  • Berant, Jonathan
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2023
Volume
11
Pages
284-299
ISSN
2307-387X (ISSN-L)
OpenAlex
W4353004366
Cited by
42
References
50
Citations by year
2026: 5 · 2025: 11 · 2024: 11 · 2023: 15
FWCI
3.72
Type
journal article
Language
en
Open access
Locations
  • Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
European Commission
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
efficient transformers, chunking, pre-trained language models, quadratic complexity, natural language understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.