Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Open accessAbstract
Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input. This phenomenon has been known as the lost-in-the-middle problem. In this work, we make three contributions. First, we set out to understand the factors that cause this phenomenon. In doing so, we establish a connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs exhibit a U-shaped attention bias where the tokens at the beginning and at the end of its input receive higher attention, regardless of their relevance. Second, we mitigate this positional bias through a calibration mechanism, found-in-the-middle, that allows the model to attend to contexts faithfully according to their relevance, even though when they are in the middle. Third, we show found-in-the-middle not only achieves better performance in locating relevant information within a long context, but also eventually leads to improved retrieval-augmented generation (RAG) performance across various tasks, outperforming existing methods by up to 15 percentage points. These findings open up future directions in understanding LLM attention bias and its potential consequences.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
10 of 15 cited sources checked · 6 supported · 0 flagged · accuracy not yet measured
- Interpretationcitation
The U-shaped attention bias correlates with the U-shaped RAG performance observed in prior literature.
“This correlates well with the U-shaped RAG performance observed in prior literature (Liu et al., 2023).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Liu et al., 2023 source
- Premisecitation
Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input.
“Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input.”
- Also stated in: § 1 Introduction · plain
“LLMs struggle to locate relevant documents when they are placed in the middle of their input prompts (Liu et al., 2023;Li et al., 2023a).”
Quote in the paperSubstantial claim1 of 2 sources checked- cites Liu et al., 2023 source
- cites Li et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Also stated in: § 1 Introduction · plain
- Premisecitation
Models preferentially use information from the beginning and end of their prompts.
“Interestingly, this focus on the beginning and end also extends to content utilization: models preferentially use information from the beginning and end of their prompts (Ravaut et al., 2023;Peysakhovich and Lerer, 2023).”
Quote in the paperSubstantial claim2 of 2 sources checked - Premisecitation
Even models specifically trained for long-context suffer from the lost-in-the-middle problem.
“Nonetheless, even models specifically trained for long-context suffer lost-in-the-middle problem (Liu et al., 2023;Li et al., 2023a).”
Quote in the paperSubstantial claim1 of 2 sources checked- cites Liu et al., 2023 source
- cites Li et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisenone given
Existing methods that manipulate input contexts improve model performance but do not fundamentally improve LLMs' underlying long-context utilization capability.
“While these existing solutions lead to improved model performances by manipulating the input contexts, they do not fundamentally improve LLMs' underlying long-context utilization capability.”
Quote in the paperSubstantial claim3 of 3 sources checked- cites Chen et al., 2023 source
The Introduction describes approaches that extend or select input context, then says they remain limited by design; the Context window scaling section identifies further limits in long-sequence attention and recurrence. This supports the concern about persistent long-context limitations, but the paper does not establish the broader claim that these methods improve performance without fundamentally improving LLMs’ underlying long-context utilization capability. (LLM, conf=0.82)
Experimental — accuracy not yet measured - cites Jiang et al., 2023
The Abstract and Section 7 (Conclusion; page unavailable) report that prompt compression improves performance, and Section 5 (Experiments; page unavailable) compares methods that manipulate input contexts. The paper does not establish that such methods fail to fundamentally improve LLMs’ underlying long-context utilization capability; it also reports that some existing compression methods perform poorly. (LLM, conf=0.82)
Experimental — accuracy not yet measured - cites Peysakhovich and Lerer, 2023 source
The Conclusion and Limitations (page unavailable) describes attention sorting as an inference-time method that modifies a model’s context to improve task performance, and suggests joint fine-tuning as a more principled solution. This supports the distinction for attention sorting, but the paper does not establish that existing context-manipulation methods generally fail to improve underlying long-context utilization capability. (LLM, conf=0.83)
Experimental — accuracy not yet measured
- Findingown data
This bias persists whether the input documents retain their original order or are randomly shuffled.
“This bias persists whether the input documents retain their original order or are randomly shuffled.”
Quote in the paperSubstantial claim - Findingown data
Calibrated attention is better correlated to the ground truth relevance of the document to a user query.
“This calibrated attention proves to be better correlated to the ground truth relevance of the document to a user query.”
Quote in the paperSubstantial claim- The authors state this as an assumption: A dummy document x_dum has a constant relevance rel(x_dum).
- Findingown data
Model performance significantly decreases when the gold document is placed within the middle of the input prompt compared to scenarios where it is placed at the beginning or end.
“Observations indicate that model performance significantly decreases when x gold is placed within the middle of the input prompt (i.e., x doc ⌊k/2⌋ ), compared to scenarios where x gold is placed at the beginning or end.”
Quote in the paperSubstantial claim - Findingown data
This dependence persists even after randomly shuffling the document order, irrespective of its relevance to the query.
“This dependence persists even after randomly shuffling the document order, irrespective of its relevance to the query.”
Quote in the paperNot a substantial claim - Findingown data
Calibrating attention leads to improvements across two popular LLMs with different context window lengths on two RAG tasks.
“We show that calibrating the attention leads to improvements across two popular LLMs with different context window lengths on two RAG tasks.”
Quote in the paperSubstantial claim - Findingown data
The model's response strongly depends on the document at the first position.
“However, quantitatively, the model's response strongly depends on the document at the first position (Figure 3).”
Quote in the paperSubstantial claim - Findingown data
The U-shaped attention pattern persists even after randomly shuffling document order.
“Crucially, the U-shaped pattern persists even after randomly shuffling document order (Figure 4, orange curve)”
Quote in the paperSubstantial claim - Findingown data
LLMs exhibit a U-shaped attention bias where tokens at the beginning and end of their input receive higher attention, regardless of their relevance.
“LLMs exhibit an U-shaped attention bias where the tokens at the beginning and at the end of its input receive higher attention, regardless of their relevance.”
- Also stated in: § 2.1 U-shaped attention bias · plain
“Figure 4 (blue curve) reveals a U-shaped attention pattern.Documents near the beginning and end of the input receive higher weights, while those in the middle receive lower weights.”
Quote in the paperSubstantial claim - Also stated in: § 2.1 U-shaped attention bias · plain
- Findingreasoning
In open-domain question answering tasks, calibrated attention outperforms popular existing approaches for ranking the relevance of retrieved documents by up to 48 Recall@3 points.
“In open-domain question answering tasks (Kwiatkowski et al., 2019), our proposed calibrated attention outperforms popular existing approaches for ranking the relevance of retrieved documents (up to 48 Recall@3 points).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Kwiatkowski et al., 2019 source
- The authors state this as an assumption: A dummy document x_dum has a constant relevance rel(x_dum).
- Findingown data
The model's response exhibits a strong bias towards the document at the first position, regardless of the gold document's location.
“Qualitatively, the model's response exhibits a strong bias towards the document at the first position, regardless of the gold document's location (Figure 2).”
Quote in the paperSubstantial claim - Findingown data
Found-in-the-middle achieves improvements over standard model generation by up to 15 percentage points on the NaturalQuestion dataset.
“Our experiments demonstrate improvements over standard model generation by up to 15 percentage point on NaturalQuestion dataset (Kwiatkowski et al., 2019).”
- Also stated in: § 4.2 Calibrated v.s. uncalibrated attention · boosted
“found-in-the-middle attention calibration consistently outperforms the uncalibrated baseline by a large margin (up to 15 percentage point (pp) improvement) across different tasks and models.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Kwiatkowski et al., 2019 source
- Also stated in: § 4.2 Calibrated v.s. uncalibrated attention · boosted
- Findingown data
Documents receiving higher attention positively correlate with them being used in the model's generation.
“In Table 1, we show that documents receiving higher attention positively correlates with them being used in the model's generation.”
Quote in the paperSubstantial claim - Findingown data
Documents near the beginning and end of the input receive higher attention weights, while those in the middle receive lower weights.
“Documents near the beginning and end of the input receive higher weights, while those in the middle receive lower weights.”
Quote in the paperSubstantial claim - Findingown data
Found-in-the-middle attention calibration consistently outperforms the uncalibrated baseline by up to 15 percentage point improvement across different tasks and models.
“In Figure 5, we see that found-in-the-middle attention calibration consistently outperforms the uncalibrated baseline by a large margin (up to 15 percentage point (pp) improvement) across different tasks and models.”
Quote in the paperSubstantial claim - Findingown data
Applying attention calibration on top of a reordering mechanism (LongLLMLingua-rk) consistently achieves the highest performance across datasets and models.
“More importantly, we see that by applying attention calibration on top of a reordering mechanism (LongLLMLingua-r k in this case), LongLLMLingua-r k with calibration consistently achieve the highest performance across datasets and models.”
Quote in the paperSubstantial claim - Findingown data
On the most challenging scenario when the gold document is placed mid-sequence, attention calibration consistently offers improvements from 6-15 pp.
“On the most challenging scenario when the gold document is placed mid-sequence, attention calibration consistently offers improvements from 6-15 pp.”
Quote in the paperSubstantial claim - Findingown data
High correlations between model attention with document relevance and positional bias support the hypothesized model.
“High correlations between model attention with document relevance and positional bias supports our hypothesized model.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Findingown data
The percentage of valid example pairs for Condition 2 is 72%.
“In Table 2, we see that the percentage of valid example pairs are decently high, 83% and 72% respectively, for both conditions, providing supports to our hypothesis.”
Quote in the paperSubstantial claim - Findingown data
Attention calibration can be applied on top of existing reordering pipelines to further improve models' performance.
“We additionally show attention calibration can be applied on top of existing reordering pipelines to further improve models' performance.”
- Also stated in: § Ethical Statement · plain
“Attention calibration can be applied on top of reordering-based methods to provide further performance boost.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Also stated in: § Ethical Statement · plain
- Findingnone given
The method introduces additional computational overhead, requiring extra O(K) model forward passes to calibrate attention at each position compared to vanilla model generation.
“Our method of calibrating positional attention bias, while effec-tive, introduces additional computational overhead.Specifically, we require extra O(K) model forward passes to calibrate attention at each position, compared to vanilla model generation.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Findingown data
There is a strong correlation where documents receiving higher attention are more likely to be used in the model's response.
“We see that there is a strong correlation where documents receiving higher attention are more likely to be used in model's response.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Findingown data
Attention calibration enables the model to find relevant context even when placed in the middle.
“Our attention calibration method enables the model to find relevant context even when placed in the middle.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Findingown data
The percentage of valid example pairs for Condition 1 is 83%.
“In Table 2, we see that the percentage of valid example pairs are decently high, 83% and 72% respectively, for both conditions, providing supports to our hypothesis.”
Quote in the paperSubstantial claim - Findingown data
The linear model Attn(x_doc, k) = rel(x_doc) + bias(k) + ε results in a Spearman's rank correlation of 0.76.
“From our study, the linear model results in decently high correlation, 0.76, suggesting its effectiveness despite the simplicity.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Model attention can be modeled as Attn(x_doc, k) = rel(x_doc) + bias(k) + ε.
- Findingown data
Intervening in early layers may lead to unstable generation.
“We find that intervening early layers may lead to unstable generation.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Findingown data
The proposed attention intervention by calibrated attention stably improves models' RAG performances compared to existing re-ordering based baselines.
“Our proposed attention intervention by calibrated attention stably improves models' RAG performances compared to existing re-ordering based baselines.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Findingown data
Average attention weights reveal a U-shaped positional bias in the model: documents at the beginning and end receive greater attention, regardless of order.
“Average attention weights reveal a Ushaped positional bias in the model.Documents at the beginning and end receive greater attention, regardless of order (gold document originally at position 10).”
- Also stated in: § Ethical Statement · plain
“We observe models exhibit U-shape attention weights favoring leading and ending contexts, regardless of their actual contents”
Quote in the paperSubstantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Also stated in: § Ethical Statement · plain
- Findingown data
Attention calibration's performance curve lies almost entirely above the vanilla baseline curve, except in 2 out of 24 cases.
“Notably, we see that attention calibration's performance curve lies almost entirely above the vanilla baseline curve (except 2 out of 24 cases), validating the effectiveness of our method in improving models' long context utilization.”
Quote in the paperSubstantial claim - Findingown data
Calibrated attention outperforms existing methods (query generation, relevance generation, vanilla attention) in ranking the relevance of retrieved contexts given a user query, as measured by Recall@3 on NaturalQuestions when gold documents are placed in the middle.
“Calibrated attention outperforms existing methods in ranking the relevance of retrieved contexts given a user query.We report Recall@3 on NaturalQuestion when gold documents are placed in the middle of input context.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Interpretationown data
The U-shaped attention bias does not depend on the documents' actual content.
“suggesting that this bias does not depend on the documents' actual content.”
Quote in the paperSubstantial claim - Interpretationreasoning
Attention calibration directly modifies the model's attention mechanism, enabling LLMs to more faithfully attend to contexts based on their relevance rather than their position.
“We mitigate the bias by attention calibration which directly modifies the model's attention mechanism, enabling LLMs to more faithfully attend to contexts based on their relevance, rather than their position.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Interpretationnone given
The paper establishes a connection between the lost-in-the-middle phenomenon and models' positional attention bias.
“In this work, we understand and address the lostin-the-middle phenomenon, by establishing a connection between the phenomenon and models' positional attention bias.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Interpretationown data
If calibrated appropriately, language models can locate relevant information even when they are hidden in the middle of the input.
“The results validate that our proposed modeling approach is effective, and that if calibrated appropriately, language models can locate relevant information even when they are hidden in the middle of the input.”
Quote in the paperSubstantial claim- The authors state this as an assumption: A dummy document x_dum has a constant relevance rel(x_dum).
- Classified as probably not a substantial claim. 11
- May combine two claims. 15
Probably not a substantial claim: may not pass S2 (checkable).
Premisenone givenThe relationship between the lost-in-the-middle problem and LLM's self-attention has been under-explored.
“However, the relationship between the lost-inthe-middle problem and LLM's self-attention has been under-explored.”
Quote in the paperNot a substantial claim1 of 2 sources checked- cites Peysakhovich and Lerer, 2023 source
The Introduction discusses the lost-in-the-middle effect and self-attention, and Section 3 examines attention weights alongside that effect. Neither passage says whether the relationship had been under-explored in prior research; searches for that characterization and for discussion of prior work did not establish it. (LLM, conf=0.72)
Experimental — accuracy not yet measured - cites He et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisereasoningSynthWiki can minimize knowledge contamination from pre-training and ensure LLMs can only use information from the provided context.
“All the documents in SynthWiki are GPT-4 generated Wikipedia paragraphs for fictional people, thus it can minimize the knowledge contamination issue from pre-training and ensure the LLMs can only use information from the provided context.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Peysakhovich and Lerer, 2023 source
May combine two claims.
Findingown dataThe model's response strongly depends on the document at the first position, and this dependence persists after randomly shuffling document order, irrespective of relevance to the query.
“Quantitatively, the model's response strongly depends on the document at the first position.This dependence persists even after randomly shuffling the document order, irrespective of its relevance to the query.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataAttention calibration is robust to the temperature term t in Eq. 5.
“In the experiments, we find attention calibration to be robust to the temperature term t in Eq. 5.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
May combine two claims.
Findingown dataCalibrated attention consistently outperforms vanilla attention by a large margin and shows superior performance compared to query generation and relevance generation re-ranking metrics.
“We see that the proposed calibrated attention consistently outperforms vanilla attention by a large margin, and also shows superior performances when compared to the other two re-ranking metrics.”
Quote in the paperSubstantial claim- The authors state this as an assumption: A dummy document x_dum has a constant relevance rel(x_dum).
May combine two claims.
Findingown dataModels do attend to relevant contexts even when placed in the middle, but are eventually distracted by leading/ending contexts.
“Models do attend to relevant contexts even when placed in the middle, but are eventually distracted by leading/ending contexts”
Quote in the paperSubstantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
May combine two claims.
Findingown dataAttention calibration improves models' context utilization/RAG performance compared to uncalibrated vanilla attention, with performance curves above vanilla in 22 of 24 cases and 6-15 point improvements when gold documents are in the middle.
“Experiments show that attention calibration improves the performance compared to its uncalibrated counterpart especially when relevant context occurs in the middle of the input.”
- Also stated in: § Ethical Statement · plain
“Attention calibration effectively improves models' context utilization ability, with its performance curves lying almost entirely above standard vanilla attention (on 22 out of 24 cases).”
- Also stated in: § Ethical Statement · plain
“On the most challenging settings where the gold documents are placed in the middle, attention calibration provides 6-15 points improvements.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Also stated in: § Ethical Statement · plain
May combine two claims.
Findingown dataThe model's response exhibits a strong bias towards the document at the first position, persisting whether input documents retain original order or are randomly shuffled.
“Qualitatively, the model's response exhibits a strong bias towards the document at the first position (red).This persists whether the input documents retain their original order (left: gold document at the 10th position) or are randomly shuffled (middle: gold document at the 13th position).”
Quote in the paperSubstantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
May combine two claims.
Findingown dataLog-linear models result in competitive rank correlation of 0.75, comparable to linear models which result in rank correlation above 0.75.
“In Appendix C, we also explore log-linear models, which results in competitive 0.75 rank correlation.”
- Also stated in: § Additional information · plain
“The log-linear model and linear are competitive to each other, which all result in rank correlation above 0.75.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Also stated in: § Additional information · plain
Classified as probably not a substantial claim.
InterpretationreasoningLost-in-the-middle happens because of the dominating force of positional bias.
“We thus conjecture that lost-in-the-middle happens because of the dominating force of positional bias.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown dataApart from U-shaped positional bias, the model exhibits an ability to prioritize relevant context.
“This contrasts with instances where x doc 10 is uncontrolled, suggesting that apart from U-shaped positional bias, the model exhibits an ability to prioritize relevant context.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataThere are two main forces driving the model attention assigned to different documents of an input prompt: where the document locates within the entire input, and the relevance of the document.
“In Sec. 2, we find that there are two main forces driving the model attention assigned to different documents of an input prompt: (a) where the document locates within the entire input, and (b) the relevance of the document.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductnone givenThe method is a pure inference method, so there is no need to do training or hyperparameter searching.
“Our method is a pure inference method, so there is no need to do training or hyperparameter searching.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone givenThe use of the datasets was consistent with their intended purpose for academic research and in accordance with their specified licensing agreements.
“We ensure that the use of these two datasets was consistent with their intended purpose for academic research and in accordance with their specified licensing agreements.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenNo personal or sensitive data was collected or utilized.
“ensuring that no personal or sensitive data was collected or utilized.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataAttention calibration can more fundamentally improve models' context utilization, providing a complementary way to re-ordering methods to further improve current RAG pipeline.
“These results suggest that attention calibration can more fundamentally improve models' context utilization, providing a complementary way to re-ordering methods to further improve current RAG pipeline.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductreasoningThe experiments use greedy decoding without any nondeterministic factor, so only need to run experiments once.
“We run our experiments with all greedy decoding without any nondeterministic factor, so we only need to run the experiments for once.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Study design and conductnone givenThe datasets should not contain any information that names or uniquely identifies individual people or offensive content.
“These two dataset should not contain any information that names or uniquely identifies individual people or offensive content.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningThe proposed method offers an efficient solution by mitigating the lost-in-the-middle problem with off-the-shelf LLMs, unlike methods requiring significant data collection and LLM tuning.
“Unlike the method, which necessitate significant investment in data collection and LLM tuning, our method offers an efficient solution by mitigating lost-in-the-middle problem with off-theshelf LLMs.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S1 (asserted), S4 (based).
Interpretationnone givenThe model attention can be modeled as Attn(x_prompt, k) = f(rel(x_doc_k), bias(k)), where f is monotonically increasing in both relevance and positional bias.
“We thus consider modeling the observable attention weights allocated to the k-th document of an input x prompt as: Attn(x prompt , k) = f (rel(x doc k ), bias(k)), (1 ) where rel(•) measures the relevance of an input document, bias(•) characterizes the positional attention bias, and f (•) is some unknown monotonically increasing function w.r.t. to both rel(x doc k ) and bias(k).”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The function f is monotonically increasing in both relevance and positional bias.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningThe paper does not definitively pinpoint the root cause of attention bias in LLMs.
“However, our work does not definitively pinpoint the root cause of attention bias in LLMs.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
The paper relies on this without support.
Re-ranking methods do not fundamentally improve LLMs' ability to utilize and capture relevant information from the provided input contexts.
“Worse, re-ranking methods do not fundamentally improve LLMs' ability to utilize and capture relevant information from the provided input contexts.”
The authors state this as an assumption.
Model attention can be modeled as Attn(x_doc, k) = rel(x_doc) + bias(k) + ε.
“Attn(x doc , k) = rel(x doc ) + bias(k) + ϵ, (2) where ϵ is a noise.”
The authors state this as an assumption.
The function f is monotonically increasing in both relevance and positional bias.
“f (•) is some unknown monotonically increasing function w.r.t. to both rel(x doc k ) and bias(k).”
The authors state this as an assumption.
A dummy document x_dum has a constant relevance rel(x_dum).
“Consider using a consistent dummy document x dum which has a constant rel(x dum )”
The authors state this as an assumption.
The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
“We proposed a simple hypothesis to model the positional attention bias, as shown in Eq. 1.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
The reported log-linear correlation is not above 0.75
MinorThe text says both model correlations exceed 0.75, but Table 4 reports exactly 0.75 for the log-linear model.
“The log-linear model and linear are competitive to each other, which all result in rank correlation above 0.75.”
“| Log-linear | 0.75 |”
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 5
- [6]6. Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference on ma- chine learning, pages 2206-2240. PMLR.
- [16]16. Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, Yibo Liu, Yuxin Liang, Hao Wang, Qianguo Sun, Songxin Zhang, Zejian Xie, et al. 2023. Never lost in the middle: Improving large language mod- els via attention strengthening question answering. arXiv e-prints, pages arXiv-2311.
- [21]21. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1-38.
- [42]42. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in huggingface. In Advances in Neural Information Processing Systems.
- [44]44. PMLR.
Cited by
Sorted by citations
Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
Jin, B., Yoon, J., Han, J., & Arik, S. O. (2024) · arXiv · doi:10.48550/arxiv.2410.05983 · Cited by 51 · see citing works
“…studies on long-context LLMs in multi-document settings (Liu et al., 2024; Shi et al., 2023) often assume a single "golden" document and random negatives, which differs from the RAG context where multiple relevant passages and hard negatives may exist (Hsieh et al., 2024b; Cuconasu et al., 2024).”
Sufficient Context: A New Lens on Retrieval Augmented Generation Systems
Joren, H., Zhang, J.-Y., Ferng, C., et al. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2411.06037 · Cited by 106
“Abstention Signals We utilize two main signals for abstention: the self-rated probabilities as in Li et al. (2024); Kadavath et al. (2022) and the sufficient context autorater.”
Eliminating Position Bias of Language Models: A Mechanistic Approach
Wang, Z.-Q., Zhang, H., Li, X.-E., et al. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2407.01100 · Cited by 88
“There are many solutions to mitigate position bias (e.g., data augmentation and training [13, 44], content resorting by attention value during inference [24], searching [39], calibration under relatively strong assumptions [12]).”
BackgroundSelf-citation(2025) · doi:10.14778/3750601.3750611 · Cited by 20 · see citing works
Leveraging long context in retrieval augmented language models for medical question answering
Zhang, G., Xu, Z., Jin, Q., Chen, F., Fang, Y., Liu, Y., Rousseau, J. F., Xu, Z., Lu, Z., Weng, C., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01651-w · Cited by 81 · see citing works
“Recent studies 27,28 pointed out that lost-in-middle-issue is attributed to the positional attention bias, i.e., models exhibit U-shaped attention patterns where documents at the beginning or end of the inputs receive higher attention values, regardless of their relevance.”
InfluentialFaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation
Zhang, Q.-G., Xiang, Z.-S., Xiao, Y.-L., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2506.08938 · Cited by 44
“The prompt templates applied in FaithfulRAG for self-knowledge extraction, self-context generation, self-fact extraction, and self-think are shown in the figure 6.”
Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generation
Merola, C., & Singh, J. (2025) · KEIR · doi:10.48550/arxiv.2504.19754 · Cited by 30
“Moreover, research indicates that LLMs may exhibit positional bias, performing better with information at the beginning of a document and struggling with content located in the middle or toward the end [11,16].”
Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation
Kim, T. E., & Díaz, F. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744599 · Cited by 20 · see citing works
GraphInsight: Unlocking Insights in Large Language Models for Graph Structure Understanding
Cao, Y., Han, S., Gao, Z., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2409.03258 · Cited by 27
“…LLMs across different positions within long sequences—a phenomenon known as “positional bias” (Tang et al., 2023; Hsieh et al., 2024; Zhang et al., 2024a), which mainly stems from limitations in the attention mechanism (Xiao et al., 2024) and the internal memory capacity (An et al., 2024) of LLMs.”
Spotlight Your Instructions: Instruction-following with Dynamic Attention SteeringPreprint
Venkateswaran, P., & Contractor, D. (2025) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2505.12025 · Cited by 27 · see citing works
“This can contribute to poor-instruction following, where insufficient attention is given to the instructions, and is exacerbated when models are given lengthy input contexts or multiple complex instructions (Li et al., 2024; Liu et al., 2024b).”
InfluentialLLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Ranking
Zeng, Y., Tendolkar, O., Baartmans, R., et al. (2024) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2406.00231 · Cited by 24
“Previous works [31, 15] also use aggregation to form the ranking list from pairwise comparison results.”
Xu, B., Wu, G., Li, Z., Xu, G., Zeng, H., Tong, R., & Ng, H. Y. (2025) · npj Clean Water · doi:10.1038/s41545-025-00509-8 · Cited by 23 · see citing works
“Techniques such as chunk reordering and calibration methods (e.g., Found-in-the-Middle) mitigate this bias but remain constrained by their reliance on unstructured retrieval and limited multi-hop reasoning capabilities 24 .”
GATEAU: Selecting Influential Samples for Long Context Alignment
Si, S., Zhao, H., Chen, G., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.375 · Cited by 21
“Meanwhile, we harness the attention weights averaged across different decoder layers and attention heads to thoroughly model how the LLM utilizes the long input contexts during the response generation (Hsieh et al., 2024).”
Sculptor: Empowering LLMs with Cognitive Agency via Active Context Management
Li, M., Xu, L.-H., Tan, Q.-T., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2508.04664 · Cited by 20
“Moreover, calibrations like Found in the Middle [Hsieh et al., 2024b] reduce—but do not eliminate—positional bias; recent evaluations [Tian et al., 2025] find that performance still degrades significantly when the distance between relevant information pieces increases, as irrelevant information…”
Distance between Relevant Information Pieces Causes Bias in Long-Context LLMsPreprint
Tian, R., Li, Y., Fu, Y., Deng, S., Luo, Q., Qian, C., Wang, S., Cong, X., Zhang, Z., Wu, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.14641
“Recent studies have shown that the position of relevant information significantly affects the performance of long-context LLMs (Liu et al., 2024; Lei et al., 2023; * Equal Contribution.”
AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generation
Fang, Y.-X., Sun, T., Shi, Y.-L., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2503.10720 · Cited by 19
“Although attention is central to LLMs, it suffers from dilution in handling long contexts (Hsieh et al., 2024).”
LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs
Chen, J., Wu, J., Xu, Y., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2503.02502 · Cited by 18
“No-tably, recent studies (Xiao et al., 2024; Hsieh et al., 2024) have shown that initial and recent tokens often receive disproportionate attention weights, suggesting a predominance of the beginning and local dependencies.”
Wang, Y., Xiong, F., Wang, Y., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2508.15709 · Cited by 17
“…Chen et al., 2024; Lin et al., 2024) and alterations to causal masks (Wang et al., 2025b), to the manipulation of internal states, including attention re-1496 The first orca to be put in captivity was weighting (Hsieh et al., 2024; Tan et al., 2025) and hidden state manipulation (Yu et al., 2024).”
Does quantization affect models' performance on long-context tasks?
Mekala, A. R., Atmakuru, A., Song, Y., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2505.20276 · Cited by 17
“…could be potentially addressed by using long-context samples during the calibration stage of quantization (Williams and Aletras, 2023), including multilingual calibration data, and applying attention-calibration techniques like found-in-the-middle to reduce attention errors (Hsieh et al., 2024b).”
Do RAG Systems Really Suffer From Positional Bias?
Cuconasu, F., Filice, S., Horowitz, G., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1422 · Cited by 17
“While previous work has established the existence of positional bias in LLMs (Liu et al., 2024; Hsieh et al., 2024; Hutter et al., 2025 systematic investigation of the impact of positional bias on distracting passages, analyzing their interactions with relevant content.”
Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs
Paulsen, N. (2026) · Advances in Artificial Intelligence and Machine Learning · doi:10.54364/aaiml.2026.61268 · Cited by 13 · see citing works
“Patterns in attention have been shown to improve data retrieval but are outside the scope of this experiment [19, 31].”
KeySG: Hierarchical Keyframe-Based 3D Scene Graphs
Werby, A., Rotondi, D., Scaparro, F., et al. (2025) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra57385.2026.11697466 · Cited by 15
“Even if the graph fits within the context window, LLMs suffer from attentional biases and a “lost in the middle" problem [8], where performance degrades as the model is distracted by the vast amount of task-irrelevant information present in the prompt.”
AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory ManagementPreprint
Wen, R., Li, H., Xiao, C., & Zhang, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.07398
“In addition, verbose context significantly degrades an agent’s decision-making capabilities by diluting LLM’s attention [30, 20].”
MMPB: It's Time for Multi-Modal Personalization
Kim, J., Kim, W., Park, W., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2509.22820 · Cited by 14
“As a result, regardless of the prompting methods, models often forget concepts located near the midpoint, reflecting the “lost-in-the-middle” effect [89, 57, 37, 103] and indicating a positional bias in attention.”
Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarization
Yang, J., Yoon, S., Chang, H., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2505.15291 · Cited by 14
“5 GPT-4o mini Mistral”
Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?Preprint
Roberts, J., Han, K., & Albanie, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.05000 · Cited by 1 · see citing works
“Similar behaviour is reported in some subsequent works (Xu et al., 2023; An et al., 2024; Dong et al., 2023; Hsieh et al., 2024b; Laban et al., 2024) (and in some cases (Levy et al., 2024)) but others have failed to replicate the findings (Zhang et al., 2024; Song et al., 2024).”
On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems
Vladika, J., & Matthes, F. (2025) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2502.14759 · Cited by 13
“This confirms the previous findings from literature that context can get "lost in the middle" of long prompts and ignored by the reader LLM when answering the questions (Liu et al., 2024; Hsieh et al., 2024).”
Hengle, A., Bajpai, P., Dan, S., et al. (2024) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2408.10151 · Cited by 13
“The preference for starting and ending positions to retrieve relevant information indicates a potential weakness in the model’s ability to maintain effective attention throughout the input sequence (Hsieh et al., 2024b), and exploring this in the multilingual setting is an important future work.”
Memory in Large Language Models: Mechanisms, Evaluation and EvolutionPreprint
Zhang, D., Li, W., Song, K., Lü, J., Li, G., Yang, L., & Li, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.18868 · Cited by 2 · see citing works
“Lost in the Middle and Found in the Middle reveal systematic positional bias [63, 64], yet many evaluations substitute window visibility for usability.”
InfluentialALAS: A Stateful Multi-LLM Agent Framework for Disruption-Aware Planning
Chang, E. Y. (2025) · ACM eBooks · doi:10.1145/3749421.3749436 · Cited by 1 · see citing works
“ALAS excels in dynamic environments where reactive decision-making, incre mental adjustments, and local consistency deliver superior results compared to global recomputation approaches.”
On the loss of context-awareness in general instruction fine-tuning
Wang, Y., Bai, A., Peng, N., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2411.02688 · Cited by 11
“Follow-up work [12] proposes to calibrate this positional bias and mitigate the issue.”
Self-citationSammour, F., Xu, J., Wang, X., Hu, M., & Zhang, Z. (2025) · Journal of Construction Engineering and Management · doi:10.1061/jcemd4.coeng-16639 · Cited by 18 · see citing works
Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMs
Salvatore, N., Wang, H., & Zhang, Q. (2025) · arXiv.org · doi:10.48550/arxiv.2510.10276 · Cited by 10
“While much of the work on the lost-in-the-middle effect has considered it a model bias and focused on eliminating the effect altogether [Hsieh et al., 2024b, Zhang et al., 2024, Wang et al., 2024], our current work provides an alternative perspective, considering it as an emergent property under…”
Leveraging LLMs for Formal Software Requirements - Challenges and Prospects
Beg, A., O'Donoghue, D., & Monahan, R. (2025) · Artificial Intelligence and fOrmal VERification, Logic, Automata, and sYnthesis · doi:10.48550/arxiv.2507.14330 · Cited by 9
“Chain-of-Thought (CoT) prompting improves logical flow via intermediate steps [29], and like Structured CoT (SCoT) [30] it can suffer from context decay in long prompts (“lost-in-the-middle”) [31], yet zero-shot often remains competitive [32].”
Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems
Byerly, A., & Khashabi, D. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.625 · Cited by 2 · see citing works
ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMs
Li, X., Li, H., Zhou, Y., Ai, Q., & Liu, Y. (2025) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3767695.3769499
Li, J.-K., Wang, R.-Z., Ma, Y.-Z., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2510.12251 · Cited by 8
“This design intentionally leverages the model’s inherent capacity to identify and amplify information flows from edge-positioned evidence, which aligns with the “U-shaped attention bias” phenomenon [16].”
Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Data
Guan, Z., Wu, L.-K., Zhao, H.-K., et al. (2025) · International Conference on Machine Learning · doi:10.48550/arxiv.2505.02130 · Cited by 8
“Duan et al. (2024) explored the relationship between attention and sentence uncertainty, while Hsieh et al. (2024) investigated the “lost-in-the-middle” phenomenon in RAG, focusing on the interaction between retrieved document ranking and attention.”
Inference Scaling for Bridging Retrieval and Augmented Generation
Lee, Y., Hwang, S.-W., Campos, D. F., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2412.10684 · Cited by 8
Positional Bias in Long-Document Ranking: Impact, Assessment, and Mitigation
Boytsov, L., Akinpelu, D., Katyal, N., et al. (2022) · IJCNLP-AACL · doi:10.18653/v1/2025.ijcnlp-long.46 · Cited by 8
“Recent work showed that this can be an intrinsic architectural artifact such that the model gives disproportionately high attention to tokens at the beginning or end of the input context, regardless of their actual relevance, which can be partially mitigated by re-calibration (Hsieh et al., 2024).”
Unstructured Evidence Attribution for Long Context Query Focused Summarization
Wright, D., Mujahid, Z. M., Wang, L., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2502.14409 · Cited by 7
Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metric
Shvartzshnaider, Y., & Duddu, V. (2026) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2026-0062
“This is an active area of research [23, 47, 75, 77].”
Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMs
Pu, R., Li, C.-Z., Rui, H., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2410.16327 · Cited by 7
“This dual-pronged tactic amplifies the focus on the harmful content through positional (Hsieh et al., 2024) and semantic guidance, thereby enhancing the likelihood of generating harmful outputs once the harmless task is executed.”
Self-citationLearning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration
Lv, H., Gu, H., Yang, R., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2604.10150 · Cited by 6
“Lastly, calibration methods like ICR (Chen et al., 2025) and FitM (Hsieh et al., 2024) leverage internal attention weights to math-ematically decouple positional priors.”
Spectral Attention Steering for Prompt Highlighting
Li, W. W., Niu, Y.-C., Yang, Y.-X., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.01281 · Cited by 6
“(1) Similarly, positional calibration methods such as Found-in-the-Middle (Hsieh et al., 2024) subtract a baseline from the positional attention bias.”
Mitigating Conversational Inertia in Multi-Turn Agents
Wan, Y., Cao, Z., Zhang, Z.-H., Zeng, Z.-W., Shen, S.-H., Meng, C.-H., & Zhu, L.-C. (2026) · arXiv.org · doi:10.48550/arxiv.2602.03664 · Cited by 7
“Neverthe-less, these insights face challenges in long-context scenarios where LLMs exhibit positional bias (Mikhail et al., 2025; Hsieh et al., 2024b) and multi-turn performance degradation (Hong et al., 2025).”
Zhang, Q., Zhang, H.-N., Pang, L., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2601.02993 · Cited by 6
A Short Survey on Formalising Software Requirements using Large Language Models
Beg, A., O'Donoghue, D., & Monahan, R. (2025) · arXiv.org · doi:10.48550/arxiv.2506.11874 · Cited by 6
“This is because of the lost-in-the-middle problem where LLM show a U-shaped attention bias [22] and can fail to attend to information in the middle of the context window.”
Training with "Paraphrasing the Original Text" Teaches LLM to Better Retrieve in Long-Context Tasks
Yu, Y., Huang, Y.-F., Qi, Z., et al. (2023) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34767 · Cited by 6
“However, while LLMs have made remarkable progress in handling long-context tasks, as shown in many evaluations (Li et al. 2023; An et al. 2023; Wang et al. 2024), they still lack satisfactory accuracy, and often suffer from a severe “lost in the middle” problem (Liu et al. 2023) when the context is…”
Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agents
Johnson, R. T., Pain, M. D., & West, J. D. (2025) · arXiv.org · doi:10.48550/arxiv.2510.14453 · Cited by 5
“Positional biases in long contexts shape model behavior (Liu et al., 2023; Peysakhovich & Lerer, 2023; Wu et al., 2025) and recent to-kens exert disproportionate influence on next-token probability distribution (Guo & Vosoughi, 2024; Hsieh et al., 2024).”
Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions
Zhang, K., Wu, L., Yu, K., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2506.11111 · Cited by 5
“…unit NSA [212], LongLoRA [23], MoBA [109], LEX [167], ALiBi Sun et al. [167] Optimizing position embedding in inputs Random text replacement [58], Longer supervised data [7] Compressing long context to short context LongLLMLingua [69], AutoCompressor [26] Toxic Noise Prompt Attack : Prompt…”
Schuhmacher, E., Michail, A., Opitz, J., et al. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2601.16934 · Cited by 4
“We introduce the notation of segments and documents used in this work and define the two concepts: positional fairness and information retention .”
Hengle, A., Bajpai, P., Dan, S., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2504.12845 · Cited by 4
“This could involve techniques from mechanistic interpretability or studying self-attention patterns, similar to recent work like (Hsieh et al. 2024b).”
RoToR: Towards More Reliable Responses for Order-Invariant Inputs
Yoon, S., Ahn, D., Lee, Y., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2502.08662 · Cited by 4
“Technically, this line of works modify the attention mechanism by altering the positional encoding to adapt an LLM to longer contexts (Peng et al., 2023; Hsieh et al., 2024; Peysakhovich and Lerer, 2023; Chen et al., 2023; Junqing et al., 2023; Xu et al., 2023; Yu et al., 2024; Zhang et al., 2024b).”
Seed-Coder: Let the Code Model Curate Data for Itself
Seed, B., Zhang, Y., Su, J., Sun, Y., Xi, C., Xiao, X., Zheng, S., Zhang, A., Liu, K., Zan, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.03524 · Cited by 3 · see citing works
“One possible explanation for this outperformance is the positional bias in attention mechanisms [Yu et al., 2024, Hsieh et al., 2024]: in SPM mode, the beginning of the suffix and the end of the prefix are positioned at opposite ends of the input token sequence, both crucial for middle-content…”
Self-citationStepwise Informativeness Search for Efficient and Effective LLM ReasoningPreprint
Wang, S., Zhao, E., Wei, Z., & Ren, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.15335
“Moreover, LLMs tend to focus on leading and recent contexts while losing attention in the middle (Hsieh et al., 2024).”
Understanding Hallucinations in Large Visual and Language Models
Ho, Z. Y., Liang, S., & Tao, D. (2026) · ACM Computing Surveys · doi:10.1145/3811409 · Cited by 3 · see citing works
“A practical diagnostic is that these failures often attenuate substantially when the input is clarified, decomposed, or better grounded.”
Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias
Chowdhury, B. D. (2026) · arXiv.org · doi:10.48550/arxiv.2603.10123 · Cited by 3
“Simultaneously, the middle-context degradation and recency bias are frequently blamed on the distance-decay properties of relative positional encodings like RoPE, sparking massive engineering efforts to flatten this decay [4, 5].”
Understanding the Dynamics of Demonstration Conflict in In-Context Learning
Jiao, D.-F., Wang, D., & Hu, L. (2026) · arXiv.org · doi:10.48550/arxiv.2603.04464 · Cited by 3
“We also observe systematic performance variance when corrupting different positions of demonstrations, also known as the positional bias (Wang et al., 2024; Cobbina & Zhou, 2025) During evaluation, we notice a strong signal that models are misled by the injected conflict.”
InfluentialShah, S. J., & Özgür, L. (2026) · arXiv.org · doi:10.48550/arxiv.2603.00801 · Cited by 3
“Recent work on calibrating positional attention bias suggests that architectural interventions can help models attend more faithfully to relevant content regardless of position [Hsieh et al., 2024].”
Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoning
Hao, Z., Li, Z., Li, W., Liu, F.-M., Zhang, M., & Li, J. (2026) · arXiv.org · doi:10.48550/arxiv.2602.06600 · Cited by 3
“These range from model-level architectural changes and calibration methods that correct positional biases (Cheng et al., 2017; Hsieh et al., 2024), to inference-time interventions that re-weight attention or re-inject evidence to steer the model back to relevant information (Gu et al., 2024; Zou et…”
Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
Xiao, Z.-K., Wang, Z., Ma, W., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2506.08371 · Cited by 3
“Additionally, (Hsieh et al., 2024b) estimates and calibrates positional bias using an auxiliary priori dummy document.”
Self-citationSame Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language ModelsPreprint
Paruchuri, A., Koyejo, S., & Adeli, E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.26079
“Position-debiasing methods include PINE (Wang et al., 2025b), attention calibration (Hsieh et al., 2024), multimodal causal-bidirectional interpolation (Tian et al., 2025), and swap-and-vote LLM-as-judge (Zheng et al., 2023).”
Kinetic theory for Transformers and the lost-in-the-middle phenomenon
Duerinckx, M., Geshkovski, B., & Rossi, S. (2026) · arXiv.org · doi:10.48550/arxiv.2605.09213 · Cited by 2
“Avoiding the resulting U -shaped profile therefore should require modifying the positional encoding so as to weaken this monotone recency mechanism or to introduce a competing phase-sensitive effect, as done in empirical works [27, 24].”
LLM-Oriented Information Retrieval: A Denoising-First Perspective
Dai, L., Sun, L., Cao, F., Rao, Z., Yang, C., Liu, H., & Xiong, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808544
Zabihi, P., Nawara, D., Ibrahim, A., & Kashef, R. F. (2026) · Applied Sciences · doi:10.3390/app16073410 · Cited by 2 · see citing works
Inducing Sustained Creativity and Diversity in Large Language Models
Luo, Q., King, G., Puett, M. J., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.19519 · Cited by 2
“These choices exploit LLMs’ “positional bias” which places greater attention to tokens at the beginning and end of input sequence [20].”
DySCO: Dynamic Attention-Scaling Decoding for Long-Context LMs
Ye, X., Zhang, W., Yin, F., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.22175 · Cited by 2
“Other approaches impose fixed, position-dependent patterns (Hsieh et al., 2024b; Zhang et al., 2024c) to mitigate the “lost-in-the-middle” issue (Liu et al., 2024a), which is less pronounced in modern LMs (e.g., Qwen3-8B performs strongly on RULER (Yang et al., 2025)).”
InfluentialLayer-wise Positional Bias in Short-Context Language Modeling
Rahimi, M., Nouri, M., & Yaghoobzadeh, Y. (2026) · arXiv.org · doi:10.48550/arxiv.2601.04098 · Cited by 2
“Across all models, early layers dominate the largest proportion of words at all positions.”
Review of Hallucination Understanding in Large Language and Vision ModelsPreprint
Ho, Z.-H., Liang, S., & Tao, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.00034
“In LLMs, Hsieh et al. [79], Pham et al. [131] and Ravaut et al. [138] examined attention weights to reveal a persistent U-shaped distribution across sequences and token permutations.”
Reranking-based Generation for Unbiased Perspective Summarization
Ri, N., Deas, N., & McKeown, K. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2506.15925 · Cited by 2
“Similar methods have been proposed to mitigate position biases in LLMs using calibration-based (Hsieh et al., 2024) and mechanistic approaches (Ratner et al., 2023; Hu et al., 2024; Liu et al., 2024a).”
Lost in the Haystack: Smaller Needles are More Difficult for LLMs to Find
Bianchi, O., Koretsky, M. J., Willey, M., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2505.18148 · Cited by 1
“…including context compression (Jiang et al., 2024), distilling long-context information into model weights (Cao et al., 2025), attention calibration (Hsieh et al., 2024b), modified positional encodings (Zhang et al., 2024b; Zheng et al., 2024), and debiased fine-tuning (Xiong et al., 2024).”
Overwhelmed by Choice: Studying LLM Decision Making at Scale
Lin, Y.-C., Seth, A., Aravind, A., et al. (2026) · doi:10.48550/arxiv.2609.32809 · Cited by 1
“Long-Context Reasoning and Positional Bias LLMs exhibit context interference, retrieval degradation, and positional biases under long contexts [Hong et al., 2025, Shi et al., 2023, Du et al., 2025, Lee et al., 2025, Liu et al., 2024, Hsieh et al., 2024, Wu et al., 2025].”
Position Bias Undermines Preference Consistency in Listwise LLM-Based Reranking
Bito, E., Ren, Y.-L., & He, E. (2026) · Proceedings of the 20th ACM Conference on Recommender Systems · doi:10.1145/3773078.3831801 · Cited by 1
“Prompt formatting and the ordering of in-context examples can affect model outputs [25], and related positional effects have been observed in question answering, retrieval, and long-context reasoning [14, 16].”
TransMem: Transforming Hidden States into Memory for Large Language Models
Lei, H., Liu, J.-M., Chen, Y.-R., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2607.29032 · Cited by 1
“However, model predictions can be sensitive to the position of relevant evidence, and long-context reasoning often degrades when useful information is distant from the generation point (Liu et al. 2024; Hsieh et al. 2024).”
Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering
Liu, J.-Y., Gu, J., Chen, S.-J., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.16494 · Cited by 1
“…Code and protocol: https://github.com/WeMWish/ lost_at_the_end_code position dependence is a property of long-context reading, with follow-ups attributing it to attention-bias asymmetry (Hsieh et al., 2024), showing direction varies by model family (Hutter et al., 2025), and reporting that Qwen-2.”
Self-citationThe First Drop of Ink: Nonlinear Impact of Misleading Information in Long-Context Reasoning
Gao, M., Chen, Z.-H., & Huang, K.-H. (2026) · arXiv.org · doi:10.48550/arxiv.2605.10828 · Cited by 1
“This limitation persists across various scenarios (Lee et al., 2025; Gao et al., 2024), motivating follow-up studies to develop methods for mitigating positional bias (Hsieh et al., 2024a; Zhang et al., 2024b; Wang et al., 2025).”
Purifying Multimodal Retrieval: Fragment-Level Evidence Selection for RAG
Wang, X., Wang, Z., Huang, C., Liu, C., Zeng, K., Sheng, Q. Z., & Yao, L. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809692
Self-citationWho Benefits from RAG? The Role of Exposure, Utility and Attribution Bias
Dehghan, M., & McDonald, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_19
“Following prior work [12,16], we assume uniform likelihood across retrieved documents to be seen by the LLM. Formally, in RAG systems, the exposure of the document d iq ∈ D q is defined as e ( d iq ) = 1 .”
Wu, Z., & Gartner, G. (2026) · arXiv.org · doi:10.48550/arxiv.2603.20578 · Cited by 1
“Cartographic displacement repositions map features to resolve visual conflicts caused by finite spatial resolution; context displacement repositions information to compensate for non-uniform attention salience [Liu et al., 2024, Wu et al., 2025, Hsieh et al., 2024].”
Decoding the Human Factor: High Fidelity Behavioral Prediction for Strategic Foresight
Yellin, B., Ezra, E., Foreman, M., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.17222 · Cited by 1
“These issues are exacerbated in long-context settings, where transformer models can underutilize information placed in the middle of long inputs (“lost in the middle”) (Liu et al., 2023; Hsieh et al., 2024).”
Document Reconstruction Unlocks Scalable Long-Context RLVR
Xiao, Y., Wang, L., Deng, Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.08237 · Cited by 1
“Models frequently suffer from the well-documented “lost in the middle” phenomenon (Liu et al., 2024), where information in the center of a long prompt is ignored, or they struggle to maintain the logical consistency of a narrative over tokens (Hsieh et al., 2024b,a; Du et al., 2025).”
Qiang, Z., Zhao, S., Wang, H., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2512.13109 · Cited by 1
“The factors influencing the calculation of attention can be represented by Equation 3 (Hsieh et al., 2024). where function f ( · ) represents the attention computation between a specific token and the last token.”
InfluentialMitigating Coordinate Prediction Bias from Positional Encoding Failures
Tao, X., Wang, Y.-W., Cai, Y.-J., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2510.22102 · Cited by 1
“High-* resolution inputs make this harder: token and compute costs rise, and larger spatial extents increase pixel–patch misalignment, leading to coordinate drift (Li et al., 2025; Gao et al., 2024; Hsieh et al., 2024; Yen et al., 2024).”
Gold Panning: Turning Positional Bias into Signal for Multi-Document LLM Reasoning
Byerly, A., & Khashabi, D. (2025) · arXiv.org · doi:10.48550/arxiv.2510.09770 · Cited by 1
“White-box meth-ods suppress bias by modifying model internals, including alternative positional encodings (Zhang et al., 2024) and attention manipulations (Peysakhovich & Lerer, 2023; Hsieh et al., 2024; Wang et al., 2024).”
Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Models
Zhu, Y., Bai, X.-F., Chen, K.-H., et al. (2025) · doi:10.48550/arxiv.2509.21984 · Cited by 1
“By combining powerful vision encoders with large language models (LLMs), these systems are able to integrate information from both modalities and perform complex reasoning.”
InfluentialSelf-Review Framework for Enhancing Instruction Following Capability of LLM
Park, S. (2025) · arXiv.org · doi:10.48550/arxiv.2507.05598 · Cited by 1
“Previous studies have shown that excessive input length can degrade LLM performance due to information loss or reinforcement of unintended biases[14, 15, 16, 17].”
Small Encoders Can Rival Large Decoders in Detecting Groundedness
Abbes, I., Prato, G., Fournier, Q., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2506.21288 · Cited by 1
“However, given that RAG leads to very long contexts, the “lost in the middle” problem [Hsieh et al., 2024, Liu et al., 2023] appears, where models struggle to use information positioned in the middle of long input sequences.”
Layer-Specific Scaling of Positional Encodings for Superior Long-Context Modeling
Wang, Z.-H., Ding, Y.-R., Lv, C., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2503.04355 · Cited by 1
“Hsieh et al. (2024) formulates a positional bias equation, rescaling the attention score matrix to address the model’s bias toward specific positions.”
Self-citationEND: Early Noise Dropping for Efficient and Effective Context Denoising
Jin, H.-Y., Chen, P., Yang, J.-F., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2502.18915 · Cited by 1
Self-citationSi, S., Zhao, H., Chen, G., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2410.15633 · Cited by 1
“Meanwhile, we harness the attention weights averaged across different decoder layers and attention heads to thoroughly model how the LLM utilizes the long input contexts during the response generation (Hsieh et al., 2024).”
Weighted Memory Tree: Remembering What Matters for Long-Horizon LLM Agents
Dao, Q., Kathalkar, P., & Eaton, K. (2026) · arXiv · doi:10.48550/arxiv.2608.20631
“…studies have shown that language models struggle to utilize relevant information within lengthy inputs, with performance often degrading well below the nominal context-window limit (Liu et al., 2024; Levy et al., 2024; Fraga, 2024; He et al., 2024; Hsieh et al., 2024; Tian et al., 2025).”
Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay
Luo, Y.-P., & Chatziafratis, V. (2026) · doi:10.48550/arxiv.2610.05564
“As a score, this attention re-ranks passages in ICR (Chen et al., 2025) and QRHead (Zhang et al., 2025b), ranks documents once calibrated for position (Found in the Middle; Hsieh et al., 2024), decides which cached tokens to keep (SnapKV; Li et al., 2024b), and, taken from a small model, which tokens a larger model re…”
TPBench: A Turning-Point Benchmark for Dialogue Compression
Park, M., Yoon, S., & Lim, H. (2026) · doi:10.48550/arxiv.2610.02736
“Liu et al. (2024a) show that accuracy is highest when the relevant passage lies at either end of the input and lowest in the middle, and Hsieh et al. (2024) trace part of that pattern to positional attention bias.”
When Context Changes: Understanding Update Failures in LLMs
Guo, J.-Y., Fang, Y.-C., Gu, S., et al. (2026) · doi:10.48550/arxiv.2609.38866
Brevik, A., Jerpseth, H., & Lafontan, S. R. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1904452
MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compression
Wan, K., Wang, Y.-F., Lai, L.-H., et al. (2026) · doi:10.48550/arxiv.2609.27380
“Context order is known to affect how effectively language models use long inputs (Hsieh et al., 2024; Zhang et al., 2024; Tang et al., 2024).”
Making Large Vision–Language Models Better Few-Shot Learners
Zhang, C., Liu, F., Xu, Y., An, Y., Cao, S., & Deng, C. D. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3732763
“Regarding recency bias, it could be caused by the distribution of training data [15] and other architectural di ff erences [16].”
Fu, W.-J. (2026) · doi:10.48550/arxiv.2609.22939
“Work on attention calibration and position-agnostic training attacks the problem inside the model [3]; we leave the model alone and change what it is allowed to read.”
Николаевич, Г. В. (2026) · Modelirovanie, optimizaciâ i informacionnye tehnologii. · doi:10.26102/2310-6018/2026.60.9.006
- Authors
- Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., & Pfister, T.
- Affiliations
- University of Washington (US)
- Massachusetts Institute of Technology (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2406.16008
- OpenAlex
- W4400023197
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- long-context modeling, retrieval-augmented generation, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
