PreprintCited by 131Topic ModelingMultimodal Machine Learning ApplicationsNatural Language Processing Techniques

The DOI matches its DataCite record; 59 of the 59 references it cites were found in a registry or an index; 10 of the 15 sources its claims cite were checked for support.

Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization

Open access

Hsieh, Cheng-Yu, Chuang, Yung-Sung, Li, Chun-Liang, Wang, Zifeng, Le, Long T., Kumar, Abhishek, Glass, James, Ratner, Alexander, Lee, Chen-Yu, Krishna, Ranjay, Pfister, Tomas·2024·arXiv

DOI 10.48550/arxiv.2406.16008arXiv 2406.16008

Open this paper in CoChat — free account Open PDF

Abstract

Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input. This phenomenon has been known as the lost-in-the-middle problem. In this work, we make three contributions. First, we set out to understand the factors that cause this phenomenon. In doing so, we establish a connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs exhibit a U-shaped attention bias where the tokens at the beginning and at the end of its input receive higher attention, regardless of their relevance. Second, we mitigate this positional bias through a calibration mechanism, found-in-the-middle, that allows the model to attend to contexts faithfully according to their relevance, even though when they are in the middle. Third, we show found-in-the-middle not only achieves better performance in locating relevant information within a long context, but also eventually leads to improved retrieval-augmented generation (RAG) performance across various tasks, outperforming existing methods by up to 15 percentage points. These findings open up future directions in understanding LLM attention bias and its potential consequences.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2406.16008

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

33 Own result · 10 Own interpretation · 6 Own method · 3 Supported · 3 Partly supported · 2 No full text of the sources · 1 Couldn’t confirm · 1 No basis given · accuracy not yet measured

Every source the paper cites for this claim was checked and supports it.
In the paper
“This correlates well with the U-shaped RAG performance observed in prior literature (Liu et al., 2023).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
What the sources say
- Supported
  The Introduction reports a U-shaped performance curve: performance is highest when relevant information is at the beginning or end of the input and degrades when it is in the middle. The multi-document question-answering results (Appendix C) show the same pattern with retrieved search results. The comparison with an attention bias is the citing authors’ own framing, not a finding attributed to Liu et al. (LLM, conf=0.91)
  cites Liu et al., 2023 source
boosted
correlational
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input.”
- Also stated in: § 1 Introduction · plain
  “LLMs struggle to locate relevant documents when they are placed in the middle of their input prompts (Liu et al., 2023;Li et al., 2023a).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Abstract
What the sources say
- Supported
  The Abstract reports that performance significantly degrades when relevant information is in the middle of a long context, even for explicitly long-context models. This supports the citing sentence’s point about models designed for long inputs. (LLM, conf=0.92)
  cites Liu et al., 2023 source
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Li et al., 2023a
plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“Interestingly, this focus on the beginning and end also extends to content utilization: models preferentially use information from the beginning and end of their prompts (Ravaut et al., 2023;Peysakhovich and Lerer, 2023).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
What the sources say
- Supported
  Section 3.2 reports that sentences from the first and last 10% of the input are much more represented in LLM summaries than sentences elsewhere. Section 4's controlled experiments likewise find a performance drop when salient information is placed away from the first or final position. The claim is a fair paraphrase of these findings about summarization prompts. (LLM, conf=0.93)
  cites Ravaut et al., 2023 source
- Supported
  Section 3 reports that accuracy varies with the relevant document’s position and describes a preference for information at the beginning and end of the context. It qualifies this pattern: in longer contexts, the beginning-of-context preference weakens and recency bias predominates. (LLM, conf=0.91)
  cites Peysakhovich and Lerer, 2023 source
plain
direct
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Nonetheless, even models specifically trained for long-context suffer lost-in-the-middle problem (Liu et al., 2023;Li et al., 2023a).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Long-context utilization in language models.
What the sources say
- Supported
  The Abstract states that performance degrades when relevant information is in the middle of long contexts, even for explicitly long-context models. The Introduction also reports that extended-context models are not necessarily better at using their input context. (LLM, conf=0.94)
  cites Liu et al., 2023 source
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Li et al., 2023a
plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While these existing solutions lead to improved model performances by manipulating the input contexts, they do not fundamentally improve LLMs' underlying long-context utilization capability.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Long-context utilization in language models.
What the sources say
- Partly supported
  The Introduction describes approaches that extend or select input context, then says they remain limited by design; the Context window scaling section identifies further limits in long-sequence attention and recurrence. This supports the concern about persistent long-context limitations, but the paper does not establish the broader claim that these methods improve performance without fundamentally improving LLMs’ underlying long-context utilization capability. (LLM, conf=0.82)
  cites Chen et al., 2023 sourceExperimental — accuracy not yet measured
- Partly supported
  The Abstract and Section 7 (Conclusion; page unavailable) report that prompt compression improves performance, and Section 5 (Experiments; page unavailable) compares methods that manipulate input contexts. The paper does not establish that such methods fail to fundamentally improve LLMs’ underlying long-context utilization capability; it also reports that some existing compression methods perform poorly. (LLM, conf=0.82)
  cites Jiang et al., 2023Experimental — accuracy not yet measured
- Partly supported
  The Conclusion and Limitations (page unavailable) describes attention sorting as an inference-time method that modifies a model’s context to improve task performance, and suggests joint fine-tuning as a more principled solution. This supports the distinction for attention sorting, but the paper does not establish that existing context-manipulation methods generally fail to improve underlying long-context utilization capability. (LLM, conf=0.83)
  cites Peysakhovich and Lerer, 2023 sourceExperimental — accuracy not yet measured
plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This bias persists whether the input documents retain their original order or are randomly shuffled.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2.1 U-shaped attention bias
plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This calibrated attention proves to be better correlated to the ground truth relevance of the document to a user query.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
plain
correlational
- The authors state this as an assumption: A dummy document x_dum has a constant relevance rel(x_dum).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observations indicate that model performance significantly decreases when x gold is placed within the middle of the input prompt (i.e., x doc ⌊k/2⌋ ), compared to scenarios where x gold is placed at the beginning or end.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2 Positional attention bias overpowers mid-sequence context
hedged
direct
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This dependence persists even after randomly shuffling the document order, irrespective of its relevance to the query.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2 Positional attention bias overpowers mid-sequence context
plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We show that calibrating the attention leads to improvements across two popular LLMs with different context window lengths on two RAG tasks.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
plain
direct
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, quantitatively, the model's response strongly depends on the document at the first position (Figure 3).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2.1 U-shaped attention bias
boosted
correlational
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Crucially, the U-shaped pattern persists even after randomly shuffling document order (Figure 4, orange curve)”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2.1 U-shaped attention bias
plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLMs exhibit an U-shaped attention bias where the tokens at the beginning and at the end of its input receive higher attention, regardless of their relevance.”
- Also stated in: § 2.1 U-shaped attention bias · plain
  “Figure 4 (blue curve) reveals a U-shaped attention pattern.Documents near the beginning and end of the input receive higher weights, while those in the middle receive lower weights.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Abstract
plain
correlational
In the paper
“In open-domain question answering tasks (Kwiatkowski et al., 2019), our proposed calibrated attention outperforms popular existing approaches for ranking the relevance of retrieved documents (up to 48 Recall@3 points).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Kwiatkowski et al., 2019 source
plain
direct
- The authors state this as an assumption: A dummy document x_dum has a constant relevance rel(x_dum).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Qualitatively, the model's response exhibits a strong bias towards the document at the first position, regardless of the gold document's location (Figure 2).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2.1 U-shaped attention bias
boosted
correlational
In the paper
“Our experiments demonstrate improvements over standard model generation by up to 15 percentage point on NaturalQuestion dataset (Kwiatkowski et al., 2019).”
- Also stated in: § 4.2 Calibrated v.s. uncalibrated attention · boosted
  “found-in-the-middle attention calibration consistently outperforms the uncalibrated baseline by a large margin (up to 15 percentage point (pp) improvement) across different tasks and models.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Kwiatkowski et al., 2019 source
boosted
direct
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Table 1, we show that documents receiving higher attention positively correlates with them being used in the model's generation.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Observation 2: Model prioritizes highlyweighted documents for generation.
plain
correlational
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Documents near the beginning and end of the input receive higher weights, while those in the middle receive lower weights.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2.1 U-shaped attention bias
plain
correlational
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Figure 5, we see that found-in-the-middle attention calibration consistently outperforms the uncalibrated baseline by a large margin (up to 15 percentage point (pp) improvement) across different tasks and models.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 4.2 Calibrated v.s. uncalibrated attention
plain
direct
n = 24
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More importantly, we see that by applying attention calibration on top of a reordering mechanism (LongLLMLingua-r k in this case), LongLLMLingua-r k with calibration consistently achieve the highest performance across datasets and models.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 4.3 Attention calibration in practice
plain
direct
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the most challenging scenario when the gold document is placed mid-sequence, attention calibration consistently offers improvements from 6-15 pp.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 4.2 Calibrated v.s. uncalibrated attention
plain
direct
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“High correlations between model attention with document relevance and positional bias supports our hypothesized model.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
plain
correlational
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Table 2, we see that the percentage of valid example pairs are decently high, 83% and 72% respectively, for both conditions, providing supports to our hypothesis.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Condition 2:
plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We additionally show attention calibration can be applied on top of existing reordering pipelines to further improve models' performance.”
- Also stated in: § Ethical Statement · plain
  “Attention calibration can be applied on top of reordering-based methods to provide further performance boost.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 6 Discussion
plain
direct
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our method of calibrating positional attention bias, while effec-tive, introduces additional computational overhead.Specifically, we require extra O(K) model forward passes to calibrate attention at each position, compared to vanilla model generation.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Computational overhead.
plain
direct
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see that there is a strong correlation where documents receiving higher attention are more likely to be used in model's response.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
boosted
correlational
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our attention calibration method enables the model to find relevant context even when placed in the middle.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
plain
direct
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Table 2, we see that the percentage of valid example pairs are decently high, 83% and 72% respectively, for both conditions, providing supports to our hypothesis.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Condition 2:
plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From our study, the linear model results in decently high correlation, 0.76, suggesting its effectiveness despite the simplicity.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Condition 2:
hedged
direct
- The authors state this as an assumption: Model attention can be modeled as Attn(x_doc, k) = rel(x_doc) + bias(k) + ε.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that intervening early layers may lead to unstable generation.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
hedged
conditional
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our proposed attention intervention by calibrated attention stably improves models' RAG performances compared to existing re-ordering based baselines.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plain
direct
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Average attention weights reveal a Ushaped positional bias in the model.Documents at the beginning and end receive greater attention, regardless of order (gold document originally at position 10).”
- Also stated in: § Ethical Statement · plain
  “We observe models exhibit U-shape attention weights favoring leading and ending contexts, regardless of their actual contents”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
plain
correlational
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, we see that attention calibration's performance curve lies almost entirely above the vanilla baseline curve (except 2 out of 24 cases), validating the effectiveness of our method in improving models' long context utilization.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 4.2 Calibrated v.s. uncalibrated attention
plain
direct
n = 24
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Calibrated attention outperforms existing methods in ranking the relevance of retrieved contexts given a user query.We report Recall@3 on NaturalQuestion when gold documents are placed in the middle of input context.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
plain
direct
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting that this bias does not depend on the documents' actual content.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2.1 U-shaped attention bias
hedged
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We mitigate the bias by attention calibration which directly modifies the model's attention mechanism, enabling LLMs to more faithfully attend to contexts based on their relevance, rather than their position.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 6 Discussion
plain
direct
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this work, we understand and address the lostin-the-middle phenomenon, by establishing a connection between the phenomenon and models' positional attention bias.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 6 Discussion
plain
correlational
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results validate that our proposed modeling approach is effective, and that if calibrated appropriately, language models can locate relevant information even when they are hidden in the middle of the input.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 3.2 Disentangling positional attention bias
plain
conditional
- The authors state this as an assumption: A dummy document x_dum has a constant relevance rel(x_dum).

Probably not a substantial claim. 11
It may state two separate claims as one. 15

The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, the relationship between the lost-inthe-middle problem and LLM's self-attention has been under-explored.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Long-context utilization in language models.
What the sources say
- Could not verify
  The Introduction discusses the lost-in-the-middle effect and self-attention, and Section 3 examines attention weights alongside that effect. Neither passage says whether the relationship had been under-explored in prior research; searches for that characterization and for discussion of prior work did not establish it. (LLM, conf=0.72)
  cites Peysakhovich and Lerer, 2023 sourceExperimental — accuracy not yet measured
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites He et al., 2023
plain
Probably not a substantial claim. It may be too vague to check against evidence.
Every source the paper cites for this claim was checked and supports it.
In the paper
“All the documents in SynthWiki are GPT-4 generated Wikipedia paragraphs for fictional people, thus it can minimize the knowledge contamination issue from pre-training and ensure the LLMs can only use information from the provided context.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
What the sources say
- Supported
  Section 2 says SynthWiki uses GPT-4 to generate one-paragraph Wikipedia-style articles about fictional people and was designed to minimize pre-training knowledge contamination. The Introduction says that, because the task data is generated, models cannot use prior knowledge to help answer its questions and must rely on the supplied documents. (LLM, conf=0.91)
  cites Peysakhovich and Lerer, 2023 source
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Quantitatively, the model's response strongly depends on the document at the first position.This dependence persists even after randomly shuffling the document order, irrespective of its relevance to the query.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
boosted
direct
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the experiments, we find attention calibration to be robust to the temperature term t in Eq. 5.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plain
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see that the proposed calibrated attention consistently outperforms vanilla attention by a large margin, and also shows superior performances when compared to the other two re-ranking metrics.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 3.2 Disentangling positional attention bias
plain
direct
It may state two separate claims as one.
- The authors state this as an assumption: A dummy document x_dum has a constant relevance rel(x_dum).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Models do attend to relevant contexts even when placed in the middle, but are eventually distracted by leading/ending contexts”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
plain
direct
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experiments show that attention calibration improves the performance compared to its uncalibrated counterpart especially when relevant context occurs in the middle of the input.”
- Also stated in: § Ethical Statement · plain
  “Attention calibration effectively improves models' context utilization ability, with its performance curves lying almost entirely above standard vanilla attention (on 22 out of 24 cases).”
- Also stated in: § Ethical Statement · plain
  “On the most challenging settings where the gold documents are placed in the middle, attention calibration provides 6-15 points improvements.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 6 Discussion
plain
direct
n = 24
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Qualitatively, the model's response exhibits a strong bias towards the document at the first position (red).This persists whether the input documents retain their original order (left: gold document at the 10th position) or are randomly shuffled (middle: gold document at the 13th position).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
boosted
correlational
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Appendix C, we also explore log-linear models, which results in competitive 0.75 rank correlation.”
- Also stated in: § Additional information · plain
  “The log-linear model and linear are competitive to each other, which all result in rank correlation above 0.75.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
plain
correlational
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We thus conjecture that lost-in-the-middle happens because of the dominating force of positional bias.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Observation 2: Model prioritizes highlyweighted documents for generation.
hedged
direct
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This contrasts with instances where x doc 10 is uncontrolled, suggesting that apart from U-shaped positional bias, the model exhibits an ability to prioritize relevant context.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2.2 Does attention favor relevant context?
hedged
correlational
Single case
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Sec. 2, we find that there are two main forces driving the model attention assigned to different documents of an input prompt: (a) where the document locates within the entire input, and (b) the relevance of the document.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 3.1 Two main factors in model attention
plain
direct
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our method is a pure inference method, so there is no need to do training or hyperparameter searching.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plain
direct
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We ensure that the use of these two datasets was consistent with their intended purpose for academic research and in accordance with their specified licensing agreements.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plain
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“ensuring that no personal or sensitive data was collected or utilized.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
plain
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results suggest that attention calibration can more fundamentally improve models' context utilization, providing a complementary way to re-ordering methods to further improve current RAG pipeline.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 4.3 Attention calibration in practice
hedged
direct
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We run our experiments with all greedy decoding without any nondeterministic factor, so we only need to run the experiments for once.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“These two dataset should not contain any information that names or uniquely identifies individual people or offensive content.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plain
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike the method, which necessitate significant investment in data collection and LLM tuning, our method offers an efficient solution by mitigating lost-in-the-middle problem with off-theshelf LLMs.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Long-context utilization in language models.
plain
direct
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We thus consider modeling the observable attention weights allocated to the k-th document of an input x prompt as: Attn(x prompt , k) = f (rel(x doc k ), bias(k)), (1 ) where rel(•) measures the relevance of an input document, bias(•) characterizes the positional attention bias, and f (•) is some unknown monotonically increasing function w.r.t. to both rel(x doc k ) and bias(k).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Our hypothesis.
plain
direct
Probably not a substantial claim. The authors may not put it forward as true. The paper may give no reason why it holds.
- The authors state this as an assumption: The function f is monotonically increasing in both relevance and positional bias.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, our work does not definitively pinpoint the root cause of attention bias in LLMs.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Computational overhead.
plain
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).

21 of 21

The paper relies on this without support.
Re-ranking methods do not fundamentally improve LLMs' ability to utilize and capture relevant information from the provided input contexts.
“Worse, re-ranking methods do not fundamentally improve LLMs' ability to utilize and capture relevant information from the provided input contexts.”
The authors state this as an assumption.
Model attention can be modeled as Attn(x_doc, k) = rel(x_doc) + bias(k) + ε.
“Attn(x doc , k) = rel(x doc ) + bias(k) + ϵ, (2) where ϵ is a noise.”
The authors state this as an assumption.
The function f is monotonically increasing in both relevance and positional bias.
“f (•) is some unknown monotonically increasing function w.r.t. to both rel(x doc k ) and bias(k).”
The authors state this as an assumption.
A dummy document x_dum has a constant relevance rel(x_dum).
“Consider using a consistent dummy document x dum which has a constant rel(x dum )”
The authors state this as an assumption.
The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
“We proposed a simple hypothesis to model the positional attention bias, as shown in Eq. 1.”

5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
- § Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
  “The log-linear model and linear are competitive to each other, which all result in rank correlation above 0.75.”
- § Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
  “| Log-linear | 0.75 |”

1 of 1

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 5

[6]6. Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference on ma- chine learning, pages 2206-2240. PMLR.
[16]16. Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, Yibo Liu, Yuxin Liang, Hao Wang, Qianguo Sun, Songxin Zhang, Zejian Xie, et al. 2023. Never lost in the middle: Improving large language mod- els via attention strengthening question answering. arXiv e-prints, pages arXiv-2311.
[21]21. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1-38.
[42]42. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in huggingface. In Advances in Neural Information Processing Systems.
[44]44. PMLR.

Sorted by citations

Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
Jin, B., Yoon, J., Han, J., & Arik, S. O. (2024) · arXiv · doi:10.48550/arxiv.2410.05983 · Cited by 51
“…studies on long-context LLMs in multi-document settings (Liu et al., 2024; Shi et al., 2023) often assume a single "golden" document and random negatives, which differs from the RAG context where multiple relevant passages and hard negatives may exist (Hsieh et al., 2024b; Cuconasu et al., 2024).”
Sufficient Context: A New Lens on Retrieval Augmented Generation SystemsPreprint
Joren, H., Zhang, J., Ferng, C.-S., Juan, D.-C., Taly, A., & Rashtchian, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.06037 · Cited by 50
“Abstention Signals We utilize two main signals for abstention: the self-rated probabilities as in Li et al. (2024); Kadavath et al. (2022) and the sufficient context autorater.”
Eliminating Position Bias of Language Models: A Mechanistic ApproachPreprint
Wang, Z., Zhang, H., Li, X., Huang, K.-H., Han, C., Ji, S., Kakade, S. M., Peng, H., & Ji, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.01100 · Cited by 44
“There are many solutions to mitigate position bias (e.g., data augmentation and training [13, 44], content resorting by attention value during inference [24], searching [39], calibration under relatively strong assumptions [12]).”
BackgroundSelf-citation
Untitled work
(2025) · doi:10.14778/3750601.3750611 · Cited by 20
Leveraging long context in retrieval augmented language models for medical question answering
Zhang, G., Xu, Z., Jin, Q., Chen, F., Fang, Y., Liu, Y., Rousseau, J. F., Xu, Z., Lu, Z., Weng, C., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01651-w · Cited by 81
“Recent studies 27,28 pointed out that lost-in-middle-issue is attributed to the positional attention bias, i.e., models exhibit U-shaped attention patterns where documents at the beginning or end of the inputs receive higher attention values, regardless of their relevance.”
Influential

See all 131

Authors: Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., & Pfister, T.
Affiliations: University of Washington (US)
Massachusetts Institute of Technology (US)
Google (United States) (US)
ORCID: Hsieh, Cheng-Yu
Chuang, Yung-Sung
Wang, Zifeng
Le, Long T.
Glass, James
Krishna, Ranjay
Venue: arXiv
Publisher: arXiv
Published: 2024
ISSN: 2331-8422 (ISSN-L)
DOI: 10.48550/arxiv.2406.16008
arXiv: 2406.16008
OpenAlex: W4400023197
Cited by: 0
References: 0
Type: posted content
Language: en
Open access: green
arxiv.org
Licence: arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations: arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords: long-context modeling, retrieval-augmented generation, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks