The DOI matches its DataCite record; 59 of the 59 references it cites were found in a registry or an index; 10 of the 15 sources its claims cite were checked for support.
Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Open accessAbstract
Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input. This phenomenon has been known as the lost-in-the-middle problem. In this work, we make three contributions. First, we set out to understand the factors that cause this phenomenon. In doing so, we establish a connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs exhibit a U-shaped attention bias where the tokens at the beginning and at the end of its input receive higher attention, regardless of their relevance. Second, we mitigate this positional bias through a calibration mechanism, found-in-the-middle, that allows the model to attend to contexts faithfully according to their relevance, even though when they are in the middle. Third, we show found-in-the-middle not only achieves better performance in locating relevant information within a long context, but also eventually leads to improved retrieval-augmented generation (RAG) performance across various tasks, outperforming existing methods by up to 15 percentage points. These findings open up future directions in understanding LLM attention bias and its potential consequences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
33 Own result · 10 Own interpretation · 6 Own method · 3 Supported · 3 Partly supported · 2 No full text of the sources · 1 Couldn’t confirm · 1 No basis given · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“This correlates well with the U-shaped RAG performance observed in prior literature (Liu et al., 2023).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
What the sources say
- Supportedcites Liu et al., 2023 source
The Introduction reports a U-shaped performance curve: performance is highest when relevant information is at the beginning or end of the input and degrades when it is in the middle. The multi-document question-answering results (Appendix C) show the same pattern with retrieved search results. The comparison with an attention bias is the citing authors’ own framing, not a finding attributed to Liu et al. (LLM, conf=0.91)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input.”
- Also stated in: § 1 Introduction · plain
“LLMs struggle to locate relevant documents when they are placed in the middle of their input prompts (Liu et al., 2023;Li et al., 2023a).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Abstract
What the sources say
- Supportedcites Liu et al., 2023 source
The Abstract reports that performance significantly degrades when relevant information is in the middle of a long context, even for explicitly long-context models. This supports the citing sentence’s point about models designed for long inputs. (LLM, conf=0.92)
- Not matchedcites Li et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“Interestingly, this focus on the beginning and end also extends to content utilization: models preferentially use information from the beginning and end of their prompts (Ravaut et al., 2023;Peysakhovich and Lerer, 2023).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
What the sources say
- Supportedcites Ravaut et al., 2023 source
Section 3.2 reports that sentences from the first and last 10% of the input are much more represented in LLM summaries than sentences elsewhere. Section 4's controlled experiments likewise find a performance drop when salient information is placed away from the first or final position. The claim is a fair paraphrase of these findings about summarization prompts. (LLM, conf=0.93)
- Supportedcites Peysakhovich and Lerer, 2023 source
Section 3 reports that accuracy varies with the relevant document’s position and describes a preference for information at the beginning and end of the context. It qualifies this pattern: in longer contexts, the beginning-of-context preference weakens and recency bias predominates. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Nonetheless, even models specifically trained for long-context suffer lost-in-the-middle problem (Liu et al., 2023;Li et al., 2023a).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Long-context utilization in language models.
What the sources say
- Supportedcites Liu et al., 2023 source
The Abstract states that performance degrades when relevant information is in the middle of long contexts, even for explicitly long-context models. The Introduction also reports that extended-context models are not necessarily better at using their input context. (LLM, conf=0.94)
- Not matchedcites Li et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While these existing solutions lead to improved model performances by manipulating the input contexts, they do not fundamentally improve LLMs' underlying long-context utilization capability.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Long-context utilization in language models.
What the sources say
- Partly supportedcites Chen et al., 2023 sourceExperimental — accuracy not yet measured
The Introduction describes approaches that extend or select input context, then says they remain limited by design; the Context window scaling section identifies further limits in long-sequence attention and recurrence. This supports the concern about persistent long-context limitations, but the paper does not establish the broader claim that these methods improve performance without fundamentally improving LLMs’ underlying long-context utilization capability. (LLM, conf=0.82)
- Partly supportedcites Jiang et al., 2023Experimental — accuracy not yet measured
The Abstract and Section 7 (Conclusion; page unavailable) report that prompt compression improves performance, and Section 5 (Experiments; page unavailable) compares methods that manipulate input contexts. The paper does not establish that such methods fail to fundamentally improve LLMs’ underlying long-context utilization capability; it also reports that some existing compression methods perform poorly. (LLM, conf=0.82)
- Partly supportedcites Peysakhovich and Lerer, 2023 sourceExperimental — accuracy not yet measured
The Conclusion and Limitations (page unavailable) describes attention sorting as an inference-time method that modifies a model’s context to improve task performance, and suggests joint fine-tuning as a more principled solution. This supports the distinction for attention sorting, but the paper does not establish that existing context-manipulation methods generally fail to improve underlying long-context utilization capability. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This bias persists whether the input documents retain their original order or are randomly shuffled.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2.1 U-shaped attention bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This calibrated attention proves to be better correlated to the ground truth relevance of the document to a user query.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: A dummy document x_dum has a constant relevance rel(x_dum).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observations indicate that model performance significantly decreases when x gold is placed within the middle of the input prompt (i.e., x doc ⌊k/2⌋ ), compared to scenarios where x gold is placed at the beginning or end.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2 Positional attention bias overpowers mid-sequence context
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This dependence persists even after randomly shuffling the document order, irrespective of its relevance to the query.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2 Positional attention bias overpowers mid-sequence context
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We show that calibrating the attention leads to improvements across two popular LLMs with different context window lengths on two RAG tasks.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 11
- It may state two separate claims as one. 15
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, the relationship between the lost-inthe-middle problem and LLM's self-attention has been under-explored.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Long-context utilization in language models.
What the sources say
- Could not verifycites Peysakhovich and Lerer, 2023 sourceExperimental — accuracy not yet measured
The Introduction discusses the lost-in-the-middle effect and self-attention, and Section 3 examines attention weights alongside that effect. Neither passage says whether the relationship had been under-explored in prior research; searches for that characterization and for discussion of prior work did not establish it. (LLM, conf=0.72)
- Not matchedcites He et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“All the documents in SynthWiki are GPT-4 generated Wikipedia paragraphs for fictional people, thus it can minimize the knowledge contamination issue from pre-training and ensure the LLMs can only use information from the provided context.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
What the sources say
- Supportedcites Peysakhovich and Lerer, 2023 source
Section 2 says SynthWiki uses GPT-4 to generate one-paragraph Wikipedia-style articles about fictional people and was designed to minimize pre-training knowledge contamination. The Introduction says that, because the task data is generated, models cannot use prior knowledge to help answer its questions and must rely on the supplied documents. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Quantitatively, the model's response strongly depends on the document at the first position.This dependence persists even after randomly shuffling the document order, irrespective of its relevance to the query.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the experiments, we find attention calibration to be robust to the temperature term t in Eq. 5.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see that the proposed calibrated attention consistently outperforms vanilla attention by a large margin, and also shows superior performances when compared to the other two re-ranking metrics.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 3.2 Disentangling positional attention bias
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: A dummy document x_dum has a constant relevance rel(x_dum).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Models do attend to relevant contexts even when placed in the middle, but are eventually distracted by leading/ending contexts”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experiments show that attention calibration improves the performance compared to its uncalibrated counterpart especially when relevant context occurs in the middle of the input.”
- Also stated in: § Ethical Statement · plain
“Attention calibration effectively improves models' context utilization ability, with its performance curves lying almost entirely above standard vanilla attention (on 22 out of 24 cases).”
- Also stated in: § Ethical Statement · plain
“On the most challenging settings where the gold documents are placed in the middle, attention calibration provides 6-15 points improvements.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 6 Discussion
plaindirectn = 24- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Also stated in: § Ethical Statement · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Qualitatively, the model's response exhibits a strong bias towards the document at the first position (red).This persists whether the input documents retain their original order (left: gold document at the 10th position) or are randomly shuffled (middle: gold document at the 13th position).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Appendix C, we also explore log-linear models, which results in competitive 0.75 rank correlation.”
- Also stated in: § Additional information · plain
“The log-linear model and linear are competitive to each other, which all result in rank correlation above 0.75.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Also stated in: § Additional information · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We thus conjecture that lost-in-the-middle happens because of the dominating force of positional bias.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Observation 2: Model prioritizes highlyweighted documents for generation.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This contrasts with instances where x doc 10 is uncontrolled, suggesting that apart from U-shaped positional bias, the model exhibits an ability to prioritize relevant context.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2.2 Does attention favor relevant context?
hedgedcorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Sec. 2, we find that there are two main forces driving the model attention assigned to different documents of an input prompt: (a) where the document locates within the entire input, and (b) the relevance of the document.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 3.1 Two main factors in model attention
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our method is a pure inference method, so there is no need to do training or hyperparameter searching.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We ensure that the use of these two datasets was consistent with their intended purpose for academic research and in accordance with their specified licensing agreements.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“ensuring that no personal or sensitive data was collected or utilized.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results suggest that attention calibration can more fundamentally improve models' context utilization, providing a complementary way to re-ordering methods to further improve current RAG pipeline.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 4.3 Attention calibration in practice
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We run our experiments with all greedy decoding without any nondeterministic factor, so we only need to run the experiments for once.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“These two dataset should not contain any information that names or uniquely identifies individual people or offensive content.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike the method, which necessitate significant investment in data collection and LLM tuning, our method offers an efficient solution by mitigating lost-in-the-middle problem with off-theshelf LLMs.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Long-context utilization in language models.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We thus consider modeling the observable attention weights allocated to the k-th document of an input x prompt as: Attn(x prompt , k) = f (rel(x doc k ), bias(k)), (1 ) where rel(•) measures the relevance of an input document, bias(•) characterizes the positional attention bias, and f (•) is some unknown monotonically increasing function w.r.t. to both rel(x doc k ) and bias(k).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Our hypothesis.
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper may give no reason why it holds.
- The authors state this as an assumption: The function f is monotonically increasing in both relevance and positional bias.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, our work does not definitively pinpoint the root cause of attention bias in LLMs.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Computational overhead.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
The paper relies on this without support.
Re-ranking methods do not fundamentally improve LLMs' ability to utilize and capture relevant information from the provided input contexts.
“Worse, re-ranking methods do not fundamentally improve LLMs' ability to utilize and capture relevant information from the provided input contexts.”
The authors state this as an assumption.
Model attention can be modeled as Attn(x_doc, k) = rel(x_doc) + bias(k) + ε.
“Attn(x doc , k) = rel(x doc ) + bias(k) + ϵ, (2) where ϵ is a noise.”
The authors state this as an assumption.
The function f is monotonically increasing in both relevance and positional bias.
“f (•) is some unknown monotonically increasing function w.r.t. to both rel(x doc k ) and bias(k).”
The authors state this as an assumption.
A dummy document x_dum has a constant relevance rel(x_dum).
“Consider using a consistent dummy document x dum which has a constant rel(x dum )”
The authors state this as an assumption.
The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
“We proposed a simple hypothesis to model the positional attention bias, as shown in Eq. 1.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
“The log-linear model and linear are competitive to each other, which all result in rank correlation above 0.75.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
“| Log-linear | 0.75 |”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 5
- [6]6. Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference on ma- chine learning, pages 2206-2240. PMLR.
- [16]16. Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, Yibo Liu, Yuxin Liang, Hao Wang, Qianguo Sun, Songxin Zhang, Zejian Xie, et al. 2023. Never lost in the middle: Improving large language mod- els via attention strengthening question answering. arXiv e-prints, pages arXiv-2311.
- [21]21. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1-38.
- [42]42. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in huggingface. In Advances in Neural Information Processing Systems.
- [44]44. PMLR.
Cited by
Sorted by citations
Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
Jin, B., Yoon, J., Han, J., & Arik, S. O. (2024) · arXiv · doi:10.48550/arxiv.2410.05983 · Cited by 51
“…studies on long-context LLMs in multi-document settings (Liu et al., 2024; Shi et al., 2023) often assume a single "golden" document and random negatives, which differs from the RAG context where multiple relevant passages and hard negatives may exist (Hsieh et al., 2024b; Cuconasu et al., 2024).”
Sufficient Context: A New Lens on Retrieval Augmented Generation SystemsPreprint
Joren, H., Zhang, J., Ferng, C.-S., Juan, D.-C., Taly, A., & Rashtchian, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.06037 · Cited by 50
“Abstention Signals We utilize two main signals for abstention: the self-rated probabilities as in Li et al. (2024); Kadavath et al. (2022) and the sufficient context autorater.”
Eliminating Position Bias of Language Models: A Mechanistic ApproachPreprint
Wang, Z., Zhang, H., Li, X., Huang, K.-H., Han, C., Ji, S., Kakade, S. M., Peng, H., & Ji, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.01100 · Cited by 44
“There are many solutions to mitigate position bias (e.g., data augmentation and training [13, 44], content resorting by attention value during inference [24], searching [39], calibration under relatively strong assumptions [12]).”
BackgroundSelf-citation(2025) · doi:10.14778/3750601.3750611 · Cited by 20
Leveraging long context in retrieval augmented language models for medical question answering
Zhang, G., Xu, Z., Jin, Q., Chen, F., Fang, Y., Liu, Y., Rousseau, J. F., Xu, Z., Lu, Z., Weng, C., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01651-w · Cited by 81
“Recent studies 27,28 pointed out that lost-in-middle-issue is attributed to the positional attention bias, i.e., models exhibit U-shaped attention patterns where documents at the beginning or end of the inputs receive higher attention values, regardless of their relevance.”
InfluentialFaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation
Zhang, Q.-G., Xiang, Z.-S., Xiao, Y.-L., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2506.08938 · Cited by 44
“The prompt templates applied in FaithfulRAG for self-knowledge extraction, self-context generation, self-fact extraction, and self-think are shown in the figure 6.”
Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented GenerationPreprint
Merola, C., & Singh, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.19754 · Cited by 30
“Moreover, research indicates that LLMs may exhibit positional bias, performing better with information at the beginning of a document and struggling with content located in the middle or toward the end [11,16].”
Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation
Kim, T. E., & Díaz, F. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744599 · Cited by 32
GraphInsight: Unlocking Insights in Large Language Models for Graph Structure Understanding
Cao, Y., Han, S., Gao, Z., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2409.03258 · Cited by 27
“…LLMs across different positions within long sequences—a phenomenon known as “positional bias” (Tang et al., 2023; Hsieh et al., 2024; Zhang et al., 2024a), which mainly stems from limitations in the attention mechanism (Xiao et al., 2024) and the internal memory capacity (An et al., 2024) of LLMs.”
Spotlight Your Instructions: Instruction-following with Dynamic Attention SteeringPreprint
Venkateswaran, P., & Contractor, D. (2025) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2505.12025 · Cited by 27
“This can contribute to poor-instruction following, where insufficient attention is given to the instructions, and is exacerbated when models are given lengthy input contexts or multiple complex instructions (Li et al., 2024; Liu et al., 2024b).”
InfluentialLLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based RankingPreprint
Zeng, Y., Tendolkar, O., Baartmans, R., Wu, Q., Chen, L., & Wang, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.00231 · Cited by 27
“Previous works [31, 15] also use aggregation to form the ranking list from pairwise comparison results.”
Xu, B., Wu, G., Li, Z., Xu, G., Zeng, H., Tong, R., & Ng, H. Y. (2025) · npj Clean Water · doi:10.1038/s41545-025-00509-8 · Cited by 24
“Techniques such as chunk reordering and calibration methods (e.g., Found-in-the-Middle) mitigate this bias but remain constrained by their reliance on unstructured retrieval and limited multi-hop reasoning capabilities 24 .”
GATEAU: Selecting Influential Samples for Long Context Alignment
Si, S., Zhao, H., Chen, G., Li, Y., Luo, K., Lv, C., An, K., Qi, F., Chang, B., & Sun, M. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.375 · Cited by 24
“Meanwhile, we harness the attention weights averaged across different decoder layers and attention heads to thoroughly model how the LLM utilizes the long input contexts during the response generation (Hsieh et al., 2024).”
Sculptor: Empowering LLMs with Cognitive Agency via Active Context ManagementPreprint
Li, M., Xu, L., Tan, Q., Ma, L., Cao, T., & Liu, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.04664 · Cited by 14
“Moreover, calibrations like Found in the Middle [Hsieh et al., 2024b] reduce—but do not eliminate—positional bias; recent evaluations [Tian et al., 2025] find that performance still degrades significantly when the distance between relevant information pieces increases, as irrelevant information…”
Distance between Relevant Information Pieces Causes Bias in Long-Context LLMsPreprint
Tian, R., Li, Y., Fu, Y., Deng, S., Luo, Q., Qian, C., Wang, S., Cong, X., Zhang, Z., Wu, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.14641
“Recent studies have shown that the position of relevant information significantly affects the performance of long-context LLMs (Liu et al., 2024; Lei et al., 2023; * Equal Contribution.”
AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented GenerationPreprint
Fang, Y., Sun, T., Shi, Y., & Gu, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.10720 · Cited by 45
“Although attention is central to LLMs, it suffers from dilution in handling long contexts (Hsieh et al., 2024).”
LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs
Chen, J., Wu, J., Xu, Y., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2503.02502 · Cited by 18
“No-tably, recent studies (Xiao et al., 2024; Hsieh et al., 2024) have shown that initial and recent tokens often receive disproportionate attention weights, suggesting a predominance of the beginning and local dependencies.”
Wang, Y., Xiong, F., Wang, Y., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2508.15709 · Cited by 17
“…Chen et al., 2024; Lin et al., 2024) and alterations to causal masks (Wang et al., 2025b), to the manipulation of internal states, including attention re-1496 The first orca to be put in captivity was weighting (Hsieh et al., 2024; Tan et al., 2025) and hidden state manipulation (Yu et al., 2024).”
Does quantization affect models' performance on long-context tasks?
Mekala, A. R., Atmakuru, A., Song, Y., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2505.20276 · Cited by 17
“…could be potentially addressed by using long-context samples during the calibration stage of quantization (Williams and Aletras, 2023), including multilingual calibration data, and applying attention-calibration techniques like found-in-the-middle to reduce attention errors (Hsieh et al., 2024b).”
Do RAG Systems Really Suffer From Positional Bias?
Cuconasu, F., Filice, S., Horowitz, G., Maarek, Y., & Silvestri, F. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1422 · Cited by 12
“While previous work has established the existence of positional bias in LLMs (Liu et al., 2024; Hsieh et al., 2024; Hutter et al., 2025 systematic investigation of the impact of positional bias on distracting passages, analyzing their interactions with relevant content.”
Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs
Paulsen, N. (2026) · Advances in Artificial Intelligence and Machine Learning · doi:10.54364/aaiml.2026.61268 · Cited by 28
“Patterns in attention have been shown to improve data retrieval but are outside the scope of this experiment [19, 31].”
KeySG: Hierarchical Keyframe-Based 3D Scene Graphs
Werby, A., Rotondi, D., Scaparro, F., & Arras, K. O. (2026) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57385.2026.11697466 · Cited by 13
“Even if the graph fits within the context window, LLMs suffer from attentional biases and a “lost in the middle" problem [8], where performance degrades as the model is distracted by the vast amount of task-irrelevant information present in the prompt.”
AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory ManagementPreprint
Wen, R., Li, H., Xiao, C., & Zhang, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.07398
“In addition, verbose context significantly degrades an agent’s decision-making capabilities by diluting LLM’s attention [30, 20].”
MMPB: It's Time for Multi-Modal Personalization
Kim, J., Kim, W., Park, W., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2509.22820 · Cited by 14
“As a result, regardless of the prompting methods, models often forget concepts located near the midpoint, reflecting the “lost-in-the-middle” effect [89, 57, 37, 103] and indicating a positional bias in attention.”
Hallucinate at the Last in Long Response Generation: A Case Study on Long Document SummarizationPreprint
Yang, J., Yoon, S., Chang, H., Kim, B., & Lee, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.15291 · Cited by 12
“5 GPT-4o mini Mistral”
Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?Preprint
Roberts, J., Han, K., & Albanie, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.05000 · Cited by 10
“Similar behaviour is reported in some subsequent works (Xu et al., 2023; An et al., 2024; Dong et al., 2023; Hsieh et al., 2024b; Laban et al., 2024) (and in some cases (Levy et al., 2024)) but others have failed to replicate the findings (Zhang et al., 2024; Song et al., 2024).”
On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems
Vladika, J., & Matthes, F. (2025) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2502.14759 · Cited by 13
“This confirms the previous findings from literature that context can get "lost in the middle" of long prompts and ignored by the reader LLM when answering the questions (Liu et al., 2024; Hsieh et al., 2024).”
Hengle, A., Bajpai, P., Dan, S., et al. (2024) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2408.10151 · Cited by 13
“The preference for starting and ending positions to retrieve relevant information indicates a potential weakness in the model’s ability to maintain effective attention throughout the input sequence (Hsieh et al., 2024b), and exploring this in the multilingual setting is an important future work.”
Memory in Large Language Models: Mechanisms, Evaluation and EvolutionPreprint
Zhang, D., Li, W.-D., Song, K., Lu, J., Li, G., Yang, L., & Li, S. (2025) · arXiv.org · doi:10.48550/arxiv.2509.18868 · Cited by 13
“Lost in the Middle and Found in the Middle reveal systematic positional bias [63, 64], yet many evaluations substitute window visibility for usability.”
InfluentialALAS: A Stateful Multi-LLM Agent Framework for Disruption-Aware Planning
Chang, E. Y. (2025) · ACM eBooks · doi:10.1145/3749421.3749436 · Cited by 1
“ALAS excels in dynamic environments where reactive decision-making, incre mental adjustments, and local consistency deliver superior results compared to global recomputation approaches.”
On the loss of context-awareness in general instruction fine-tuning
Wang, Y., Bai, A., Peng, N., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2411.02688 · Cited by 11
“Follow-up work [12] proposes to calibrate this positional bias and mitigate the issue.”
Self-citationSammour, F., Xu, J., Wang, X., Hu, M., & Zhang, Z. (2025) · Journal of Construction Engineering and Management · doi:10.1061/jcemd4.coeng-16639 · Cited by 30
Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMs
Salvatore, N., Wang, H., & Zhang, Q. (2025) · arXiv.org · doi:10.48550/arxiv.2510.10276 · Cited by 10
“While much of the work on the lost-in-the-middle effect has considered it a model bias and focused on eliminating the effect altogether [Hsieh et al., 2024b, Zhang et al., 2024, Wang et al., 2024], our current work provides an alternative perspective, considering it as an emergent property under…”
Leveraging LLMs for Formal Software Requirements - Challenges and Prospects
Beg, A., O'Donoghue, D., & Monahan, R. (2025) · Artificial Intelligence and fOrmal VERification, Logic, Automata, and sYnthesis · doi:10.48550/arxiv.2507.14330 · Cited by 9
“Chain-of-Thought (CoT) prompting improves logical flow via intermediate steps [29], and like Structured CoT (SCoT) [30] it can suffer from context decay in long prompts (“lost-in-the-middle”) [31], yet zero-shot often remains competitive [32].”
Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems
Byerly, A., & Khashabi, D. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.625 · Cited by 8
ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMs
Li, X., Li, H., Zhou, Y., Ai, Q., & Liu, Y. (2025) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3767695.3769499
Li, J.-K., Wang, R.-Z., Ma, Y.-Z., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2510.12251 · Cited by 8
“This design intentionally leverages the model’s inherent capacity to identify and amplify information flows from edge-positioned evidence, which aligns with the “U-shaped attention bias” phenomenon [16].”
Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Data
Guan, Z., Wu, L.-K., Zhao, H.-K., et al. (2025) · International Conference on Machine Learning · doi:10.48550/arxiv.2505.02130 · Cited by 8
“Duan et al. (2024) explored the relationship between attention and sentence uncertainty, while Hsieh et al. (2024) investigated the “lost-in-the-middle” phenomenon in RAG, focusing on the interaction between retrieved document ranking and attention.”
Inference Scaling for Bridging Retrieval and Augmented Generation
Lee, Y., Hwang, S.-W., Campos, D. F., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2412.10684 · Cited by 8
Positional Bias in Long-Document Ranking: Impact, Assessment, and Mitigation
Boytsov, L., Akinpelu, D., Katyal, N., et al. (2022) · IJCNLP-AACL · doi:10.18653/v1/2025.ijcnlp-long.46 · Cited by 8
“Recent work showed that this can be an intrinsic architectural artifact such that the model gives disproportionately high attention to tokens at the beginning or end of the input context, regardless of their actual relevance, which can be partially mitigated by re-calibration (Hsieh et al., 2024).”
Unstructured Evidence Attribution for Long Context Query Focused Summarization
Wright, D., Mujahid, Z. M., Wang, L., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2502.14409 · Cited by 7
Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metric
Shvartzshnaider, Y., & Duddu, V. (2026) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2026-0062 · Cited by 23
“This is an active area of research [23, 47, 75, 77].”
Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMs
Pu, R., Li, C.-Z., Rui, H., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2410.16327 · Cited by 7
“This dual-pronged tactic amplifies the focus on the harmful content through positional (Hsieh et al., 2024) and semantic guidance, thereby enhancing the likelihood of generating harmful outputs once the harmless task is executed.”
Self-citationLearning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration
Lv, H., Gu, H., Yang, R., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2604.10150 · Cited by 6
“Lastly, calibration methods like ICR (Chen et al., 2025) and FitM (Hsieh et al., 2024) leverage internal attention weights to math-ematically decouple positional priors.”
Spectral Attention Steering for Prompt HighlightingPreprint
Li, W. W., Niu, Y., Yang, Y., Li, K., Ma, T., & Cohen, S. B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.01281 · Cited by 10
“(1) Similarly, positional calibration methods such as Found-in-the-Middle (Hsieh et al., 2024) subtract a baseline from the positional attention bias.”
Mitigating Conversational Inertia in Multi-Turn Agents
Wan, Y., Cao, Z., Zhang, Z.-H., Zeng, Z.-W., Shen, S.-H., Meng, C.-H., & Zhu, L.-C. (2026) · arXiv.org · doi:10.48550/arxiv.2602.03664 · Cited by 7
“Neverthe-less, these insights face challenges in long-context scenarios where LLMs exhibit positional bias (Mikhail et al., 2025; Hsieh et al., 2024b) and multi-turn performance degradation (Hong et al., 2025).”
Zhang, Q., Zhang, H.-N., Pang, L., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2601.02993 · Cited by 6
A Short Survey on Formalising Software Requirements using Large Language Models
Beg, A., O'Donoghue, D., & Monahan, R. (2025) · arXiv.org · doi:10.48550/arxiv.2506.11874 · Cited by 6
“This is because of the lost-in-the-middle problem where LLM show a U-shaped attention bias [22] and can fail to attend to information in the middle of the context window.”
Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks
Yu, Y., Huang, Y., Qi, Z., & Zhou, Z. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34767 · Cited by 10
“However, while LLMs have made remarkable progress in handling long-context tasks, as shown in many evaluations (Li et al. 2023; An et al. 2023; Wang et al. 2024), they still lack satisfactory accuracy, and often suffer from a severe “lost in the middle” problem (Liu et al. 2023) when the context is…”
Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language AgentsPreprint
Johnson, R. T., Pain, M. D., & West, J. D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.14453 · Cited by 6
“Positional biases in long contexts shape model behavior (Liu et al., 2023; Peysakhovich & Lerer, 2023; Wu et al., 2025) and recent to-kens exert disproportionate influence on next-token probability distribution (Guo & Vosoughi, 2024; Hsieh et al., 2024).”
Evaluating and Improving Robustness in Large Language Models: A Survey and Future DirectionsPreprint
Zhang, K., Wu, L., Yu, K.-Y., Lv, G., & Zhang, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.11111 · Cited by 16
“…unit NSA [212], LongLoRA [23], MoBA [109], LEX [167], ALiBi Sun et al. [167] Optimizing position embedding in inputs Random text replacement [58], Longer supervised data [7] Compressing long context to short context LongLLMLingua [69], AutoCompressor [26] Toxic Noise Prompt Attack : Prompt…”
Schuhmacher, E., Michail, A., Opitz, J., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2601.16934 · Cited by 4
“We introduce the notation of segments and documents used in this work and define the two concepts: positional fairness and information retention .”
Hengle, A., Bajpai, P., Dan, S., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2504.12845 · Cited by 4
“This could involve techniques from mechanistic interpretability or studying self-attention patterns, similar to recent work like (Hsieh et al. 2024b).”
RoToR: Towards More Reliable Responses for Order-Invariant Inputs
Yoon, S., Ahn, D., Lee, Y., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2502.08662 · Cited by 4
“Technically, this line of works modify the attention mechanism by altering the positional encoding to adapt an LLM to longer contexts (Peng et al., 2023; Hsieh et al., 2024; Peysakhovich and Lerer, 2023; Chen et al., 2023; Junqing et al., 2023; Xu et al., 2023; Yu et al., 2024; Zhang et al., 2024b).”
Seed-Coder: Let the Code Model Curate Data for Itself
Seed, B., Zhang, Y., Su, J., Sun, Y., Xi, C., Xiao, X., Zheng, S., Zhang, A., Liu, K., Zan, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.03524 · Cited by 3
“One possible explanation for this outperformance is the positional bias in attention mechanisms [Yu et al., 2024, Hsieh et al., 2024]: in SPM mode, the beginning of the suffix and the end of the prefix are positioned at opposite ends of the input token sequence, both crucial for middle-content…”
Self-citationStepwise Informativeness Search for Efficient and Effective LLM ReasoningPreprint
Wang, S., Zhao, E., Wei, Z., & Ren, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.15335
“Moreover, LLMs tend to focus on leading and recent contexts while losing attention in the middle (Hsieh et al., 2024).”
Understanding Hallucinations in Large Visual and Language Models
Ho, Z. Y., Liang, S., & Tao, D. (2026) · ACM Computing Surveys · doi:10.1145/3811409 · Cited by 8
“A practical diagnostic is that these failures often attenuate substantially when the input is clarified, decomposed, or better grounded.”
Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias
Chowdhury, B. D. (2026) · arXiv.org · doi:10.48550/arxiv.2603.10123 · Cited by 3
“Simultaneously, the middle-context degradation and recency bias are frequently blamed on the distance-decay properties of relative positional encodings like RoPE, sparking massive engineering efforts to flatten this decay [4, 5].”
Understanding the Dynamics of Demonstration Conflict in In-Context LearningPreprint
Jiao, D., Wang, D., & Hu, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.04464 · Cited by 5
“We also observe systematic performance variance when corrupting different positions of demonstrations, also known as the positional bias (Wang et al., 2024; Cobbina & Zhou, 2025) During evaluation, we notice a strong signal that models are misled by the injected conflict.”
InfluentialShah, S., & Özgür, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.00801 · Cited by 4
“Recent work on calibrating positional attention bias suggests that architectural interventions can help models attend more faithfully to relevant content regardless of position [Hsieh et al., 2024].”
Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoning
Hao, Z., Li, Z., Li, W., Liu, F.-M., Zhang, M., & Li, J. (2026) · arXiv.org · doi:10.48550/arxiv.2602.06600 · Cited by 3
“These range from model-level architectural changes and calibration methods that correct positional biases (Cheng et al., 2017; Hsieh et al., 2024), to inference-time interventions that re-weight attention or re-inject evidence to steer the model back to relevant information (Gu et al., 2024; Zou et…”
Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
Xiao, Z.-K., Wang, Z., Ma, W., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2506.08371 · Cited by 3
“Additionally, (Hsieh et al., 2024b) estimates and calibrates positional bias using an auxiliary priori dummy document.”
Self-citationSame Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language ModelsPreprint
Paruchuri, A., Koyejo, S., & Adeli, E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.26079 · Cited by 3
“Position-debiasing methods include PINE (Wang et al., 2025b), attention calibration (Hsieh et al., 2024), multimodal causal-bidirectional interpolation (Tian et al., 2025), and swap-and-vote LLM-as-judge (Zheng et al., 2023).”
Kinetic theory for Transformers and the lost-in-the-middle phenomenon
Duerinckx, M., Geshkovski, B., & Rossi, S. (2026) · arXiv.org · doi:10.48550/arxiv.2605.09213 · Cited by 2
“Avoiding the resulting U -shaped profile therefore should require modifying the positional encoding so as to weaken this monotone recency mechanism or to introduce a competing phase-sensitive effect, as done in empirical works [27, 24].”
LLM-Oriented Information Retrieval: A Denoising-First Perspective
Dai, L., Sun, L., Cao, F., Rao, Z., Yang, C., Liu, H., & Xiong, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808544 · Cited by 2
Zabihi, P., Nawara, D., Ibrahim, A., & Kashef, R. F. (2026) · Applied Sciences · doi:10.3390/app16073410 · Cited by 3
Inducing Sustained Creativity and Diversity in Large Language Models
Luo, Q., King, G., Puett, M. J., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.19519 · Cited by 2
“These choices exploit LLMs’ “positional bias” which places greater attention to tokens at the beginning and end of input sequence [20].”
DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language ModelsPreprint
Ye, X., Zhang, W., Yin, F., Yen, H. H., & Chen, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.22175 · Cited by 5
“Other approaches impose fixed, position-dependent patterns (Hsieh et al., 2024b; Zhang et al., 2024c) to mitigate the “lost-in-the-middle” issue (Liu et al., 2024a), which is less pronounced in modern LMs (e.g., Qwen3-8B performs strongly on RULER (Yang et al., 2025)).”
InfluentialLayer-wise Positional Bias in Short-Context Language ModelingPreprint
Rahimi, M., Nouri, M., & Yaghoobzadeh, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.04098 · Cited by 1
“Across all models, early layers dominate the largest proportion of words at all positions.”
Review of Hallucination Understanding in Large Language and Vision ModelsPreprint
Ho, Z.-H., Liang, S., & Tao, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.00034
“In LLMs, Hsieh et al. [79], Pham et al. [131] and Ravaut et al. [138] examined attention weights to reveal a persistent U-shaped distribution across sequences and token permutations.”
Reranking-based Generation for Unbiased Perspective Summarization
Ri, N., Deas, N., & McKeown, K. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2506.15925 · Cited by 2
“Similar methods have been proposed to mitigate position biases in LLMs using calibration-based (Hsieh et al., 2024) and mechanistic approaches (Ratner et al., 2023; Hu et al., 2024; Liu et al., 2024a).”
Lost in the Haystack: Smaller Needles are More Difficult for LLMs to Find
Bianchi, O., Koretsky, M. J., Willey, M., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2505.18148 · Cited by 1
“…including context compression (Jiang et al., 2024), distilling long-context information into model weights (Cao et al., 2025), attention calibration (Hsieh et al., 2024b), modified positional encodings (Zhang et al., 2024b; Zheng et al., 2024), and debiased fine-tuning (Xiong et al., 2024).”
Overwhelmed by Choice: Studying LLM Decision Making at Scale
Lin, Y.-C., Seth, A., Aravind, A., et al. (2026) · doi:10.48550/arxiv.2609.32809 · Cited by 1
“Long-Context Reasoning and Positional Bias LLMs exhibit context interference, retrieval degradation, and positional biases under long contexts [Hong et al., 2025, Shi et al., 2023, Du et al., 2025, Lee et al., 2025, Liu et al., 2024, Hsieh et al., 2024, Wu et al., 2025].”
Position Bias Undermines Preference Consistency in Listwise LLM-Based Reranking
Bito, E., Ren, Y.-L., & He, E. (2026) · Proceedings of the 20th ACM Conference on Recommender Systems · doi:10.1145/3773078.3831801 · Cited by 1
“Prompt formatting and the ordering of in-context examples can affect model outputs [25], and related positional effects have been observed in question answering, retrieval, and long-context reasoning [14, 16].”
TransMem: Transforming Hidden States into Memory for Large Language Models
Lei, H., Liu, J.-M., Chen, Y.-R., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2607.29032 · Cited by 1
“However, model predictions can be sensitive to the position of relevant evidence, and long-context reasoning often degrades when useful information is distant from the generation point (Liu et al. 2024; Hsieh et al. 2024).”
Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering
Liu, J.-Y., Gu, J., Chen, S.-J., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.16494 · Cited by 1
“…Code and protocol: https://github.com/WeMWish/ lost_at_the_end_code position dependence is a property of long-context reading, with follow-ups attributing it to attention-bias asymmetry (Hsieh et al., 2024), showing direction varies by model family (Hutter et al., 2025), and reporting that Qwen-2.”
Self-citationThe First Drop of Ink: Nonlinear Impact of Distracting Information in Long-Context ReasoningPreprint
Gao, M., Chen, Z.-C., & Huang, K.-H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.10828 · Cited by 3
“This limitation persists across various scenarios (Lee et al., 2025; Gao et al., 2024), motivating follow-up studies to develop methods for mitigating positional bias (Hsieh et al., 2024a; Zhang et al., 2024b; Wang et al., 2025).”
Purifying Multimodal Retrieval: Fragment-Level Evidence Selection for RAG
Wang, X., Wang, Z., Huang, C., Liu, C., Zeng, K., Sheng, Q. Z., & Yao, L. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809692
Self-citationWho Benefits from RAG? The Role of Exposure, Utility and Attribution Bias
Dehghan, M., & McDonald, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_19
“Following prior work [12,16], we assume uniform likelihood across retrieved documents to be seen by the LLM. Formally, in RAG systems, the exposure of the document d iq ∈ D q is defined as e ( d iq ) = 1 .”
Wu, Z., & Gärtner, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.20578 · Cited by 1
“Cartographic displacement repositions map features to resolve visual conflicts caused by finite spatial resolution; context displacement repositions information to compensate for non-uniform attention salience [Liu et al., 2024, Wu et al., 2025, Hsieh et al., 2024].”
Decoding the Human Factor: High Fidelity Behavioral Prediction for Strategic Foresight
Yellin, B., Ezra, E., Foreman, M., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.17222 · Cited by 1
“These issues are exacerbated in long-context settings, where transformer models can underutilize information placed in the middle of long inputs (“lost in the middle”) (Liu et al., 2023; Hsieh et al., 2024).”
Document Reconstruction Unlocks Scalable Long-Context RLVR
Xiao, Y., Wang, L., Deng, Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.08237 · Cited by 1
“Models frequently suffer from the well-documented “lost in the middle” phenomenon (Liu et al., 2024), where information in the center of a long prompt is ignored, or they struggle to maintain the logical consistency of a narrative over tokens (Hsieh et al., 2024b,a; Du et al., 2025).”
Qiang, Z., Zhao, S., Wang, H., Qin, B., & Liu, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.13109
“The factors influencing the calculation of attention can be represented by Equation 3 (Hsieh et al., 2024). where function f ( · ) represents the attention computation between a specific token and the last token.”
InfluentialMitigating Coordinate Prediction Bias from Positional Encoding FailuresPreprint
Tao, X., Wang, Y., Cai, Y., Luo, Y., Han, K., & Tang, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.22102 · Cited by 2
“High-* resolution inputs make this harder: token and compute costs rise, and larger spatial extents increase pixel–patch misalignment, leading to coordinate drift (Li et al., 2025; Gao et al., 2024; Hsieh et al., 2024; Yen et al., 2024).”
Gold Panning: Turning Positional Bias into Signal for Multi-Document LLM Reasoning
Byerly, A., & Khashabi, D. (2025) · arXiv.org · doi:10.48550/arxiv.2510.09770 · Cited by 1
“White-box meth-ods suppress bias by modifying model internals, including alternative positional encodings (Zhang et al., 2024) and attention manipulations (Peysakhovich & Lerer, 2023; Hsieh et al., 2024; Wang et al., 2024).”
Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language ModelsPreprint
Zhu, Y., Bai, X., Chen, K., Xiang, Y., Pan, Y., Hou, Y., Guan, W., Yu, J., & Zhang, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.21984 · Cited by 3
“By combining powerful vision encoders with large language models (LLMs), these systems are able to integrate information from both modalities and perform complex reasoning.”
InfluentialSelf-Review Framework for Enhancing Instruction Following Capability of LLM
Park, S. (2025) · arXiv.org · doi:10.48550/arxiv.2507.05598 · Cited by 1
“Previous studies have shown that excessive input length can degrade LLM performance due to information loss or reinforcement of unintended biases[14, 15, 16, 17].”
Small Encoders Can Rival Large Decoders in Detecting Groundedness
Abbes, I., Prato, G., Fournier, Q., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2506.21288 · Cited by 1
“However, given that RAG leads to very long contexts, the “lost in the middle” problem [Hsieh et al., 2024, Liu et al., 2023] appears, where models struggle to use information positioned in the middle of long input sequences.”
Layer-Specific Scaling of Positional Encodings for Superior Long-Context Modeling
Wang, Z.-H., Ding, Y.-R., Lv, C., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2503.04355 · Cited by 1
“Hsieh et al. (2024) formulates a positional bias equation, rescaling the attention score matrix to address the model’s bias toward specific positions.”
Self-citationEND: Early Noise Dropping for Efficient and Effective Context DenoisingPreprint
Jin, H., Chen, P., Yang, J., Wang, Z., Mo, F., Zhang, J., Jiang, M., Gao, Y., Huang, B., Zhang, X., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.18915 · Cited by 3
Self-citationSi, S., Zhao, H., Chen, G., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2410.15633 · Cited by 1
“Meanwhile, we harness the attention weights averaged across different decoder layers and attention heads to thoroughly model how the LLM utilizes the long input contexts during the response generation (Hsieh et al., 2024).”
Weighted Memory Tree: Remembering What Matters for Long-Horizon LLM Agents
Dao, Q., Kathalkar, P., & Eaton, K. (2026) · arXiv · doi:10.48550/arxiv.2608.20631 · Cited by 1
“…studies have shown that language models struggle to utilize relevant information within lengthy inputs, with performance often degrading well below the nominal context-window limit (Liu et al., 2024; Levy et al., 2024; Fraga, 2024; He et al., 2024; Hsieh et al., 2024; Tian et al., 2025).”
Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention RelayPreprint
Luo, Y., & Chatziafratis, V. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.05564
“As a score, this attention re-ranks passages in ICR (Chen et al., 2025) and QRHead (Zhang et al., 2025b), ranks documents once calibrated for position (Found in the Middle; Hsieh et al., 2024), decides which cached tokens to keep (SnapKV; Li et al., 2024b), and, taken from a small model, which tokens a larger model re…”
TPBench: A Turning-Point Benchmark for Dialogue CompressionPreprint
Park, M., Yoon, S., & Lim, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.02736
“Liu et al. (2024a) show that accuracy is highest when the relevant passage lies at either end of the input and lowest in the middle, and Hsieh et al. (2024) trace part of that pattern to positional attention bias.”
When Context Changes: Understanding Update Failures in LLMsPreprint
Guo, J., Fang, Y., Gu, S., Spanos, C. J., Demmel, J. W., & Lavaei, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.38866
Brevik, A., Jerpseth, H., & Lafontan, S. R. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1904452
MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compression
Wan, K., Wang, Y.-F., Lai, L.-H., et al. (2026) · doi:10.48550/arxiv.2609.27380
“Context order is known to affect how effectively language models use long inputs (Hsieh et al., 2024; Zhang et al., 2024; Tang et al., 2024).”
Making Large Vision–Language Models Better Few-Shot Learners
Zhang, C., Liu, F., Xu, Y., An, Y., Cao, S., & Deng, C. D. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3732763
“Regarding recency bias, it could be caused by the distribution of training data [15] and other architectural di ff erences [16].”
Fu, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.22939
“Work on attention calibration and position-agnostic training attacks the problem inside the model [3]; we leave the model alone and change what it is allowed to read.”
Николаевич, Г. В. (2026) · Modelirovanie, optimizaciâ i informacionnye tehnologii. · doi:10.26102/2310-6018/2026.60.9.006
Metadata
- Authors
- Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., & Pfister, T.
- Affiliations
- University of Washington (US)
- Massachusetts Institute of Technology (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2406.16008
- OpenAlex
- W4400023197
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- long-context modeling, retrieval-augmented generation, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
