The DOI matches its DataCite record; 59 of the 59 references it cites were found in a registry or an index; 10 of the 15 sources its claims cite were checked for support.
Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Open accessAbstract
Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input. This phenomenon has been known as the lost-in-the-middle problem. In this work, we make three contributions. First, we set out to understand the factors that cause this phenomenon. In doing so, we establish a connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs exhibit a U-shaped attention bias where the tokens at the beginning and at the end of its input receive higher attention, regardless of their relevance. Second, we mitigate this positional bias through a calibration mechanism, found-in-the-middle, that allows the model to attend to contexts faithfully according to their relevance, even though when they are in the middle. Third, we show found-in-the-middle not only achieves better performance in locating relevant information within a long context, but also eventually leads to improved retrieval-augmented generation (RAG) performance across various tasks, outperforming existing methods by up to 15 percentage points. These findings open up future directions in understanding LLM attention bias and its potential consequences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
33 Own result · 10 Own interpretation · 6 Own method · 3 Supported · 3 Partly supported · 2 No full text of the sources · 1 Couldn’t confirm · 1 No basis given · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“This correlates well with the U-shaped RAG performance observed in prior literature (Liu et al., 2023).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
What the sources say
- Supportedcites Liu et al., 2023 source
The Introduction reports a U-shaped performance curve: performance is highest when relevant information is at the beginning or end of the input and degrades when it is in the middle. The multi-document question-answering results (Appendix C) show the same pattern with retrieved search results. The comparison with an attention bias is the citing authors’ own framing, not a finding attributed to Liu et al. (LLM, conf=0.91)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input.”
- Also stated in: § 1 Introduction · plain
“LLMs struggle to locate relevant documents when they are placed in the middle of their input prompts (Liu et al., 2023;Li et al., 2023a).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Abstract
What the sources say
- Supportedcites Liu et al., 2023 source
The Abstract reports that performance significantly degrades when relevant information is in the middle of a long context, even for explicitly long-context models. This supports the citing sentence’s point about models designed for long inputs. (LLM, conf=0.92)
- Not matchedcites Li et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“Interestingly, this focus on the beginning and end also extends to content utilization: models preferentially use information from the beginning and end of their prompts (Ravaut et al., 2023;Peysakhovich and Lerer, 2023).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
What the sources say
- Supportedcites Ravaut et al., 2023 source
Section 3.2 reports that sentences from the first and last 10% of the input are much more represented in LLM summaries than sentences elsewhere. Section 4's controlled experiments likewise find a performance drop when salient information is placed away from the first or final position. The claim is a fair paraphrase of these findings about summarization prompts. (LLM, conf=0.93)
- Supportedcites Peysakhovich and Lerer, 2023 source
Section 3 reports that accuracy varies with the relevant document’s position and describes a preference for information at the beginning and end of the context. It qualifies this pattern: in longer contexts, the beginning-of-context preference weakens and recency bias predominates. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Nonetheless, even models specifically trained for long-context suffer lost-in-the-middle problem (Liu et al., 2023;Li et al., 2023a).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Long-context utilization in language models.
What the sources say
- Supportedcites Liu et al., 2023 source
The Abstract states that performance degrades when relevant information is in the middle of long contexts, even for explicitly long-context models. The Introduction also reports that extended-context models are not necessarily better at using their input context. (LLM, conf=0.94)
- Not matchedcites Li et al., 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While these existing solutions lead to improved model performances by manipulating the input contexts, they do not fundamentally improve LLMs' underlying long-context utilization capability.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Long-context utilization in language models.
What the sources say
- Partly supportedcites Chen et al., 2023 sourceExperimental — accuracy not yet measured
The Introduction describes approaches that extend or select input context, then says they remain limited by design; the Context window scaling section identifies further limits in long-sequence attention and recurrence. This supports the concern about persistent long-context limitations, but the paper does not establish the broader claim that these methods improve performance without fundamentally improving LLMs’ underlying long-context utilization capability. (LLM, conf=0.82)
- Partly supportedcites Jiang et al., 2023Experimental — accuracy not yet measured
The Abstract and Section 7 (Conclusion; page unavailable) report that prompt compression improves performance, and Section 5 (Experiments; page unavailable) compares methods that manipulate input contexts. The paper does not establish that such methods fail to fundamentally improve LLMs’ underlying long-context utilization capability; it also reports that some existing compression methods perform poorly. (LLM, conf=0.82)
- Partly supportedcites Peysakhovich and Lerer, 2023 sourceExperimental — accuracy not yet measured
The Conclusion and Limitations (page unavailable) describes attention sorting as an inference-time method that modifies a model’s context to improve task performance, and suggests joint fine-tuning as a more principled solution. This supports the distinction for attention sorting, but the paper does not establish that existing context-manipulation methods generally fail to improve underlying long-context utilization capability. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This bias persists whether the input documents retain their original order or are randomly shuffled.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2.1 U-shaped attention bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This calibrated attention proves to be better correlated to the ground truth relevance of the document to a user query.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: A dummy document x_dum has a constant relevance rel(x_dum).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Observations indicate that model performance significantly decreases when x gold is placed within the middle of the input prompt (i.e., x doc ⌊k/2⌋ ), compared to scenarios where x gold is placed at the beginning or end.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2 Positional attention bias overpowers mid-sequence context
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This dependence persists even after randomly shuffling the document order, irrespective of its relevance to the query.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2 Positional attention bias overpowers mid-sequence context
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We show that calibrating the attention leads to improvements across two popular LLMs with different context window lengths on two RAG tasks.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 11
- It may state two separate claims as one. 15
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, the relationship between the lost-inthe-middle problem and LLM's self-attention has been under-explored.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Long-context utilization in language models.
What the sources say
- Could not verifycites Peysakhovich and Lerer, 2023 sourceExperimental — accuracy not yet measured
The Introduction discusses the lost-in-the-middle effect and self-attention, and Section 3 examines attention weights alongside that effect. Neither passage says whether the relationship had been under-explored in prior research; searches for that characterization and for discussion of prior work did not establish it. (LLM, conf=0.72)
- Not matchedcites He et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“All the documents in SynthWiki are GPT-4 generated Wikipedia paragraphs for fictional people, thus it can minimize the knowledge contamination issue from pre-training and ensure the LLMs can only use information from the provided context.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
What the sources say
- Supportedcites Peysakhovich and Lerer, 2023 source
Section 2 says SynthWiki uses GPT-4 to generate one-paragraph Wikipedia-style articles about fictional people and was designed to minimize pre-training knowledge contamination. The Introduction says that, because the task data is generated, models cannot use prior knowledge to help answer its questions and must rely on the supplied documents. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Quantitatively, the model's response strongly depends on the document at the first position.This dependence persists even after randomly shuffling the document order, irrespective of its relevance to the query.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the experiments, we find attention calibration to be robust to the temperature term t in Eq. 5.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see that the proposed calibrated attention consistently outperforms vanilla attention by a large margin, and also shows superior performances when compared to the other two re-ranking metrics.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 3.2 Disentangling positional attention bias
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: A dummy document x_dum has a constant relevance rel(x_dum).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Models do attend to relevant contexts even when placed in the middle, but are eventually distracted by leading/ending contexts”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experiments show that attention calibration improves the performance compared to its uncalibrated counterpart especially when relevant context occurs in the middle of the input.”
- Also stated in: § Ethical Statement · plain
“Attention calibration effectively improves models' context utilization ability, with its performance curves lying almost entirely above standard vanilla attention (on 22 out of 24 cases).”
- Also stated in: § Ethical Statement · plain
“On the most challenging settings where the gold documents are placed in the middle, attention calibration provides 6-15 points improvements.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 6 Discussion
plaindirectn = 24- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Also stated in: § Ethical Statement · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Qualitatively, the model's response exhibits a strong bias towards the document at the first position (red).This persists whether the input documents retain their original order (left: gold document at the 10th position) or are randomly shuffled (middle: gold document at the 13th position).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Appendix C, we also explore log-linear models, which results in competitive 0.75 rank correlation.”
- Also stated in: § Additional information · plain
“The log-linear model and linear are competitive to each other, which all result in rank correlation above 0.75.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
- Also stated in: § Additional information · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We thus conjecture that lost-in-the-middle happens because of the dominating force of positional bias.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Observation 2: Model prioritizes highlyweighted documents for generation.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This contrasts with instances where x doc 10 is uncontrolled, suggesting that apart from U-shaped positional bias, the model exhibits an ability to prioritize relevant context.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 2.2 Does attention favor relevant context?
hedgedcorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Sec. 2, we find that there are two main forces driving the model attention assigned to different documents of an input prompt: (a) where the document locates within the entire input, and (b) the relevance of the document.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 3.1 Two main factors in model attention
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our method is a pure inference method, so there is no need to do training or hyperparameter searching.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We ensure that the use of these two datasets was consistent with their intended purpose for academic research and in accordance with their specified licensing agreements.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“ensuring that no personal or sensitive data was collected or utilized.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Ethical Statement
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results suggest that attention calibration can more fundamentally improve models' context utilization, providing a complementary way to re-ordering methods to further improve current RAG pipeline.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > 4.3 Attention calibration in practice
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We run our experiments with all greedy decoding without any nondeterministic factor, so we only need to run the experiments for once.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“These two dataset should not contain any information that names or uniquely identifies individual people or offensive content.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike the method, which necessitate significant investment in data collection and LLM tuning, our method offers an efficient solution by mitigating lost-in-the-middle problem with off-theshelf LLMs.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Long-context utilization in language models.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We thus consider modeling the observable attention weights allocated to the k-th document of an input x prompt as: Attn(x prompt , k) = f (rel(x doc k ), bias(k)), (1 ) where rel(•) measures the relevance of an input document, bias(•) characterizes the positional attention bias, and f (•) is some unknown monotonically increasing function w.r.t. to both rel(x doc k ) and bias(k).”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Our hypothesis.
plaindirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper may give no reason why it holds.
- The authors state this as an assumption: The function f is monotonically increasing in both relevance and positional bias.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, our work does not definitively pinpoint the root cause of attention bias in LLMs.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Computational overhead.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
The paper relies on this without support.
Re-ranking methods do not fundamentally improve LLMs' ability to utilize and capture relevant information from the provided input contexts.
“Worse, re-ranking methods do not fundamentally improve LLMs' ability to utilize and capture relevant information from the provided input contexts.”
The authors state this as an assumption.
Model attention can be modeled as Attn(x_doc, k) = rel(x_doc) + bias(k) + ε.
“Attn(x doc , k) = rel(x doc ) + bias(k) + ϵ, (2) where ϵ is a noise.”
The authors state this as an assumption.
The function f is monotonically increasing in both relevance and positional bias.
“f (•) is some unknown monotonically increasing function w.r.t. to both rel(x doc k ) and bias(k).”
The authors state this as an assumption.
A dummy document x_dum has a constant relevance rel(x_dum).
“Consider using a consistent dummy document x dum which has a constant rel(x dum )”
The authors state this as an assumption.
The positional attention bias can be modeled as a simple additive combination of document relevance and position bias (Eq. 1).
“We proposed a simple hypothesis to model the positional attention bias, as shown in Eq. 1.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
“The log-linear model and linear are competitive to each other, which all result in rank correlation above 0.75.”
§ Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization > Additional information
“| Log-linear | 0.75 |”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 5
- [6]6. Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference on ma- chine learning, pages 2206-2240. PMLR.
- [16]16. Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, Yibo Liu, Yuxin Liang, Hao Wang, Qianguo Sun, Songxin Zhang, Zejian Xie, et al. 2023. Never lost in the middle: Improving large language mod- els via attention strengthening question answering. arXiv e-prints, pages arXiv-2311.
- [21]21. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1-38.
- [42]42. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in huggingface. In Advances in Neural Information Processing Systems.
- [44]44. PMLR.
Cited by
Sorted by citations
Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
Jin, B., Yoon, J., Han, J., & Arik, S. O. (2024) · arXiv · doi:10.48550/arxiv.2410.05983 · Cited by 51
“…studies on long-context LLMs in multi-document settings (Liu et al., 2024; Shi et al., 2023) often assume a single "golden" document and random negatives, which differs from the RAG context where multiple relevant passages and hard negatives may exist (Hsieh et al., 2024b; Cuconasu et al., 2024).”
Sufficient Context: A New Lens on Retrieval Augmented Generation SystemsPreprint
Joren, H., Zhang, J., Ferng, C.-S., Juan, D.-C., Taly, A., & Rashtchian, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.06037 · Cited by 50
“Abstention Signals We utilize two main signals for abstention: the self-rated probabilities as in Li et al. (2024); Kadavath et al. (2022) and the sufficient context autorater.”
Eliminating Position Bias of Language Models: A Mechanistic ApproachPreprint
Wang, Z., Zhang, H., Li, X., Huang, K.-H., Han, C., Ji, S., Kakade, S. M., Peng, H., & Ji, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.01100 · Cited by 44
“There are many solutions to mitigate position bias (e.g., data augmentation and training [13, 44], content resorting by attention value during inference [24], searching [39], calibration under relatively strong assumptions [12]).”
BackgroundSelf-citation(2025) · doi:10.14778/3750601.3750611 · Cited by 20
Leveraging long context in retrieval augmented language models for medical question answering
Zhang, G., Xu, Z., Jin, Q., Chen, F., Fang, Y., Liu, Y., Rousseau, J. F., Xu, Z., Lu, Z., Weng, C., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01651-w · Cited by 81
“Recent studies 27,28 pointed out that lost-in-middle-issue is attributed to the positional attention bias, i.e., models exhibit U-shaped attention patterns where documents at the beginning or end of the inputs receive higher attention values, regardless of their relevance.”
Influential
Metadata
- Authors
- Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., & Pfister, T.
- Affiliations
- University of Washington (US)
- Massachusetts Institute of Technology (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2406.16008
- OpenAlex
- W4400023197
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- long-context modeling, retrieval-augmented generation, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
