Lost in the Middle: How Language Models Use Long Contexts

With too much context supplied at once, the LLM is at greater risk of generating 'hallucinations'.

“With too much context supplied at once, the LLM is at greater risk of generating 'hallucinations' (or, semantically plausible but factually incorrect text) (Najork 2023; Liu et al. 2023).”

Heuristic retrieval may return too few entries, omitting crucial context, or too many, flooding the model with irrelevant information and degrading performance.

“While this extends access to past information, it also creates a fundamental retrieval challenge: heuristics may return too few entries, omitting crucial context, or too many, flooding the model with irrelevant information and degrading performance (Liu et al., 2023).”

In the domain of RAG, the standard practice often involves a singular (once) retrieval step followed by generation, which can lead to inefficiencies and sometimes is typically insufficient for complex problems demanding multi-step reasoning, as it provides a limited scope of information.

“In the domain of RAG, the standard practice often involves a singular (once) retrieval step followed by generation, which can lead to inefficiencies and sometimes is typically insufficient for complex problems demanding multi-step reasoning, as it provides a limited scope of information [105].”

Language models often exhibit a localized focus within their attention mechanisms, giving priority to nearby words and resulting in a notable deficit in context attention.

“However, as demonstrated in prior studies [19, 189, 307], language models often exhibit a localized focus within their attention mechanisms, giving priority to nearby words and resulting in a notable deficit in context attention [275].”

AI systems have been shown to rely more strongly on data at the beginning and end of the context window.

“Although AI systems are being equipped with increasingly longer context windows (which is the part of the current conversation the model can “remember”, like a working memory), they have been shown to rely stronger on data at the beginning and et the end of the window”

The U-shaped attention bias correlates with the U-shaped RAG performance observed in prior literature.

“This correlates well with the U-shaped RAG performance observed in prior literature (Liu et al., 2023).”

Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input.

“Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input.”

Even models specifically trained for long-context suffer from the lost-in-the-middle problem.

“Nonetheless, even models specifically trained for long-context suffer lost-in-the-middle problem (Liu et al., 2023;Li et al., 2023a).”

LLMs can encounter significant performance degradation due to insufficient utilization of the context, especially for information located in the middle of the long context window (lost-in-the-middle).

“Despite successfully retrieving evidence relevant to factoid queries, LLMs can encounter a significant performance degradation due to insufficient utilization of the context, especially for information located in the middle of the long context window, a notable issue known as the lost-in-the-middle phenomenon [189].”

Long-context memory can lead to biases and unrobustness in LLM's inference; specifically, previous research has shown that the positions of text segments in a long context can greatly affect their utilization, so the memory in the long-context prompt can not be treated equally and stably.

“Last but not least, it can lead to biases and unrobustness in LLM's inference. Specifically, a previous research [120] has shown that, the positions of text segments in a long context can greatly affect their utilization, so the memory in the long-context prompt can not be treated equally and stably.”