Longformer: The Long-Document Transformer

Local-window memory approaches (Alibi position encoding, sliding-window attention) also struggle on the multi-unrelated-documents question-answering task for the same reason.

“Similarly, local-window memory approaches, such as Alibi position encoding Press et al. [2021] and sliding-window attention Beltagy et al. [2020], also struggle for the same reason.”

Five principal strategies have emerged to bridge the memory gap: in-context windowing, external key-value store, graph-based episodic memory, compression-based summarisation, and web-augmented memory.

“Five principal strategies have emerged to bridge this gap:”

Sparse-attention and positional bias studies prove that models struggle to maintain strict instruction salience across multi-turn conversational updates.

“sparse-attention (Longformer, BigBird) [10, 11] and positional bias studies (ALiBi) [12] prove that while models can physically process long sequences, they struggle to maintain strict instruction salience across multi-turn conversational updates.”

Language models often exhibit a localized focus within their attention mechanisms, giving priority to nearby words and resulting in a notable deficit in context attention.

“However, as demonstrated in prior studies [19, 189, 307], language models often exhibit a localized focus within their attention mechanisms, giving priority to nearby words and resulting in a notable deficit in context attention [275].”

The Longformer-based label prediction model's performance was not competitive with the pipeline setup, likely because the label predictor struggles to identify relevant information when given full abstracts.

“We experimented with a label prediction model which encodes entire abstracts via the Longformer (Beltagy et al., 2020), and makes predictions using the document-level CLS token. Performance was not competitive with our pipeline setup, likely because the label predictor struggles to identify relevant information when given full abstracts.”

Recent progress in long-context modeling largely targets static inputs and does not address multi-turn interaction with external environments.

“Recent progress in long-context modeling largely targets static inputs ( e.g., long documents) and does not address multi-turn interaction with external environments [6, 19].”

LED extends the input length limit to 16,384 tokens.

“which extends the input length limit to 16,384 tokens.”

Special attention mechanisms such as Big Bird, LED, and LongT5 are designed to ease input limits for long-text input.

“special attention mechanisms are designed to ease the input limits for the processing of long-text input. Some examples include Big Bird (Zaheer et al., 2020), Longformer-Encoder-Decoder (LED) (Beltagy et al., 2020), and LongT5 (Guo et al., 2022).”

Directly extending the context length of transformers incurs a quadratic increase in computational time and memory cost due to the transformer architecture's self-attention mechanism.

“Directly extending the context length of transformers incurs a quadratic increase in computational time and memory cost due to the transformer architecture's self-attention mechanism, making the design of new long-context architectures a pressing research challenge (Dai et al., 2019; Kitaev et al., 2020; Beltagy et al., 2020).”

The modular KV cache construction in Prompt Cache bears resemblance to locally masked attention.

“The modular KV cache construction in Prompt Cache bears resemblance to the approximations observed in locally masked attention (Beltagy et al., 2020; Tay et al., 2023), which optimizes computations by setting a limited window for attention score calculations rather than spanning its attention across every token in its input sequence.”

LED overcomes the sentence length limit from BART.

“Because the transcript is long, LED overcomes the sentence length limit from BART.”

Big Bird, Longformer, and LongNet help extend context length for Transformers by switching from full self-attention to sparse self-attention mechanisms with linear complexity.

“Big Bird (Zaheer et al., 2020), Longformer (Beltagy et al., 2020), LongNet (Ding et al., 2023) help extend context length for Transformers by switching from full self-attention to sparse self-attention mechanisms with linear complexity.”

For large sequence lengths N, s is often set to N^{-1/2} or N^{-1} log N, resulting in Θ(N sqrt N) or Θ(N log N) IO complexity.

“For large sequence lengths 𝑁 , 𝑠 is often set to 𝑁 1 2 [11] or 𝑁 1 log 𝑁 [3, 17, 92], resulting in Θ 𝑁 𝑁 or Θ 𝑁 log 𝑁 IO complexity.”

When encoders learn wrong correlations between different parts of the training data, it could result in erroneous generation that diverges from the input.

“When encoders learn wrong correlations between different parts of the training data, it could result in erroneous generation that diverges from the input [6, 71, 142, 251].”

Models augmented with the Focus Attention Mechanism generate more faithful summaries on the BBC extreme summarization task.

“The results on the BBC extreme summarization task show that models augmented with the Focus Attention Mechanism generate more faithful summaries.”

Kosmos-2 strengthens joint modeling of images and language through self-supervised learning, allowing it to handle more complex tasks such as deep reasoning and generation of image and text.

“Kosmos-2 [210] strengthens the joint modeling of images and language through self-supervised learning, allowing it to handle more complex tasks, such as deep reasoning and generation of image and text.”