Lost in the Middle: How Language Models Use Long Contexts
Open accessThe DOI matches its Crossref record; 48 of the 59 references it cites were found in a registry or an index; 15 of the 23 sources its claims cite were checked for support.
Abstract
While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts: multi-document question answering and key-value retrieval. We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.
Overview
CoChat Summary
TL;DR
Language models’ performance on tasks requiring access to long input contexts is strongly affected by where relevant information appears, with a U-shaped curve: best at the beginning or end and worst in the middle. Extended-context models are not necessarily better at using context, and reader performance saturates before retriever recall in open-domain QA.
Key findings
- In multi-document QA, performance is highest when the answer-containing document is at the start or end of the context and degrades in the middle; GPT-3.5-Turbo can drop by more than 20%, and in 20- and 30-document settings middle-position performance is lower than its closed-book performance of 56.1% (oracle: 88.3%).
- Extended-context models are not necessarily better at using input context: when the input fits both windows, GPT-3.5-Turbo and GPT-3.5-Turbo(16K) perform nearly identically as a function of relevant-information position.
- In synthetic key-value retrieval, Claude-1.3 and Claude-1.3(100K) do nearly perfectly, while other models struggle especially with 140 or 300 key-value pairs; query-aware contextualization gives GPT-3.5-Turbo(16K) perfect performance at 300 pairs, versus worst-case 45.6% without it.
- In open-domain QA, reader performance saturates long before retriever recall: using 50 instead of 20 retrieved documents only marginally improves performance (∼1.5% for GPT-3.5-Turbo and ∼1% for Claude-1.3).
Methods
- Controlled multi-document QA on NaturalQuestions-Open: 2,655 queries whose annotated long answer is a paragraph; each context has exactly one answer-containing Wikipedia passage plus k−1 Contriever/MS-MARCO distractor passages; the paper varies answer position and context length (10, 20, 30 documents) and measures accuracy.
- Synthetic key-value retrieval: inputs are serialized JSON objects with unique random UUID key-value pairs; models must return the value for a specified key; experiments use 75, 140, and 300 key-value pairs (500 examples each), varying key position and number of pairs, with accuracy as the metric.
- Models include open MPT-30B-Instruct and LongChat-13B(16K), closed GPT-3.5-Turbo, GPT-3.5-Turbo(16K), Claude-1.3, and Claude-1.3(100K), plus additional architecture/instruction experiments with Flan-T5-XXL, Flan-UL2, MPT-30B, Llama-2, and GPT-4; an open-domain QA case study uses a Contriever retriever-reader setup.
Limitations
- Stated: the paper uses greedy decoding and leaves exploration of other decoding methods to future work; the investigations into why models are not robust are described as preliminary; GPT-4 is evaluated only on a subset of 500 multi-document QA examples because full evaluation would cost upwards of $6000.
- Stated: NaturalQuestions-Open has ambiguity, so the authors run an unambiguous subset and also test random distractors and randomized distractor order, finding similar trends.
- Evident: the controlled multi-document QA setting always contains exactly one document that answers the question, unlike open-domain QA; the synthetic key-value task removes natural language semantics; results may not generalize to all tasks, models, or decoding settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
7 Valid · 0 Not valid · 5 Not checked · 13 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Language models have become an important and flexible building block in a variety of user-facing language technologies, including conversational interfaces, search and summarization, and collaborative writing (Shuster et al., 2022;Thoppilan et al., 2022;Lee et al., 2022, inter alia).”
§ Lost in the Middle: How Language Models Use Long Contexts > 1 Introduction
What the sources say
- Partly supportedcites Shuster et al., 2022 matched to “BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage” (2022) sourceExperimental — accuracy not yet measured
Section 3.1.1 (p. 4) describes a conversational agent built on a language model that generates internet search queries, supporting conversational interfaces and search as part of dialogue. Section 2 (p. 3) mentions summarization only as related work, and the passages do not establish collaborative writing as an application of this paper’s model. (LLM, conf=0.85)
- Supportedcites Thoppilan et al., 2022 matched to “LaMDA: Language Models for Dialog Applications” (2022) source
The Abstract (p. 1) identifies LaMDA as a family of language models specialized for dialog. In this sentence's list of applications, Thoppilan et al. supports the conversational-interface example; the other cited works need not each cover every example. (LLM, conf=0.92)
- Supportedcites Lee et al., 2022 matched to “CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities” (2022) source
Lee et al. is cited alongside other works, and its contribution to the sentence is the collaborative-writing example. Section 4.2.2 describes writers receiving and selecting model-generated suggestions in a writing interface; Section 2.2.1 also describes language models as applicable to dialogue and summarization. The paper need not establish every example in the jointly cited list. (LLM, conf=0.89)
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These observations complement prior work, which found that non-instruction fine-tuned language models are biased towards recent tokens (i.e., the end of the input context; Khandelwal et al., 2018;Press et al., 2021).”
§ Lost in the Middle: How Language Models Use Long Contexts > 4.3 Effect of Instruction Fine-Tuning
What the sources say
- Partly supportedcites Khandelwal et al., 2018 matched to “Sharp nearby, fuzzy far away: How neural language models use context” (2018) sourceExperimental — accuracy not yet measured
The Introduction and Section 5 report that the studied LSTM is much more sensitive to recent word order than to distant word order, supporting a recency effect. The paper studies a standard LSTM on two datasets, however, rather than establishing this tendency across non-instruction-fine-tuned language models generally; it also finds that distant words remain relevant. (LLM, conf=0.82)
- Partly supportedcites Press et al., 2021 matched to “Shortformer: Better language modeling using shorter inputs” (2021) sourceExperimental — accuracy not yet measured
Section 3.1 (page not available in our copy) finds that very short contexts hurt performance, while increasing subsequence length from 1,024 to 3,072 tokens gives negligible improvement. That is consistent with limited benefit from more distant context, but the paper does not establish that the models are biased toward recent tokens within an input context. (LLM, conf=0.78)
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
§ Lost in the Middle: How Language Models Use Long Contexts > 2.1 Experimental Setup
What the sources say
- Supportedcites Lee et al., 2019 matched to “Latent retrieval for weakly supervised open domain question answering” (2019) source
Section 7.1 says the paper creates an open version of Natural Questions and uses it for training and evaluation. The Introduction describes the task as answering questions by retrieving evidence from an open corpus, supporting the cited use of NaturalQuestions-Open for this task. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites Kwiatkowski et al., 2019 matched to “Natural Questions: A benchmark for question answering research” (2019) source
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Following Kandpal et al. (2022) and Mallen et al. (2023), we use accuracy as our primary evaluation metric, judging whether any of the cor- rect answers (as taken from the NaturalQuestions annotations) appear in the predicted output.”
§ Lost in the Middle: How Language Models Use Long Contexts > 2.1 Experimental Setup
What the sources say
- Supportedcites Kandpal et al. (2022) matched to “Large Language Models Struggle to Learn Long-Tail Knowledge” (2022) source
Section 3 (p. 4) says the paper evaluates answers using exact match against a ground-truth answer set, and Section 3.1 (p. 4) reports QA accuracies as its evaluation results. This supports citing Kandpal et al. for using accuracy to evaluate question answering. (LLM, conf=0.90)
- Supportedcites Mallen et al. (2023) matched to “When not to trust language models: Investigating effectiveness of parametric and non-parametric memories” (2023) source
Section 3.1 (‘Focus and Task’) explicitly names accuracy as the metric and says a prediction is correct when it contains an exact match to any gold answer. This supports the evaluation approach attributed to Mallen et al.; the NaturalQuestions annotations describe the citing authors’ answer source. (LLM, conf=0.94)
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“or when language models are augmented with external information (relevant documents from a search engine, database query results, etc.; Petroni et al., 2020;Ram et al., 2023;Shi et al., 2023;Mallen et al., 2023;Schick et al., 2023, inter alia).”
§ Lost in the Middle: How Language Models Use Long Contexts > 1 Introduction
What the sources say
- Partly supportedcites Petroni et al., 2020 matched to “How Context Affects Language Models' Factual Predictions” (2020) sourceExperimental — accuracy not yet measured
The Abstract (p. 1) says the paper integrates information from a retrieval system with a pre-trained language model, and §3.4.2 (p. 5) specifies a retrieved paragraph as context. This supports augmentation with retrieved documents, but our copy does not show language models being augmented with database query results. (LLM, conf=0.84)
- Partly supportedcites Ram et al., 2023 matched to “In-Context Retrieval-Augmented Language Models” (2023) sourceExperimental — accuracy not yet measured
The Introduction describes augmenting language models with relevant documents retrieved from an external knowledge source, and Section 5 describes using a BM25 retriever to find those documents. The passages do not establish database query results as another type of external information used by this paper. (LLM, conf=0.82)
- Partly supportedcites Shi et al., 2023 matched to “REPLUG: Retrieval-augmented black-box language models” (2023) sourceExperimental — accuracy not yet measured
Section 3.1 (page unavailable) describes retrieving relevant documents from a corpus, and the Conclusion says REPLUG uses those documents to augment a language model. The paper supports the document-retrieval example, but our copy does not describe augmenting models with database query results or documents from a search engine specifically. (LLM, conf=0.82)
- Partly supported
20 more claims · 2 with their cited sources checked
Create a free account to see all 25 claims and every source verdict.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In vanilla multi-document or NIAH-based benchmarks (Kamradt, 2023; Liu et al., 2024), models perform consistently well when the needle (or gold document) appears at the very beginning or end of the context window, with minimal impact from context length.”
From NoLiMa: Long-Context Evaluation Beyond Literal Matching (2025) · cites this paper as Liu et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Even within a single long sequence, attention effectiveness degrades with distance due to attention dilution , positional encoding limitations, and token interference, leading to the well-known 'lost-in-the-middle' and context-decay phenomena (Liu et al., 2024; Press et al., 2021a).”
From MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents (2026) · cites this paper as Liu et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current large language models (LLMs) exhibit significant deficiencies in episodic memory tasks including encoding, storing, and retrieving specific information from temporally dependent events over a long period of time.”
From ARTEM: Enhancing Large Language Model Agents with Spatial-Temporal Episodic Memory (2026) · cites this paper as Liu et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent approaches to handle memory tasks in LLMs, such as in-context learning, retrieval-augmented generation (RAG), and fine-tuning, may resolve the long-term retention issues, but are still inadequate to handle tasks requiring chronological awareness of the stored information.”
From ARTEM: Enhancing Large Language Model Agents with Spatial-Temporal Episodic Memory (2026) · cites this paper as Liu et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“If not ranked at the top, the key information may be neglected by the generative module, resulting in incomplete or inaccurate responses to the user queries24.”
From Leveraging long context in retrieval augmented language models for medical question answering (2025) · cites this paper as [24]
Experimental — accuracy not yet measured
86 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 2×
“Recent improvements in hardware (e.g., faster GPUs with more memory) and algorithms (Dai et al., 2019; Dao et al., 2022; Poli et al., 2023; Rubin and Berant, 2023, inter alia ) have resulted in language models with larger context windows (e.g., 4096, 32K, and even 100K tokens), but it remains…”
BackgroundLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2020) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2007.01282 · Cited by 2,281
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 1×
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019; Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
MethodThe serial position effect of free recall.
Murdock, B. B. (1962) · Journal of Experimental Psychology · doi:10.1037/h0045106 · Cited by 1,717 · cited 1×
“The U-shaped curve we observe in this work has a connection in psychology known as the serial-position effect (Ebbinghaus, 1913; Murdock Jr, 1962), that states that in free-association recall of elements from a list, humans tend to best re-member the first and last elements of the list.”
MethodMemory: A contribution to experimental psychology.
Ebbinghaus, H. (1913) · doi:10.1037/10011-000 · Cited by 1,645 · cited 1×
“The U-shaped curve we observe in this work has a connection in psychology known as the serial-position effect (Ebbinghaus, 1913; Murdock Jr, 1962), that states that in free-association recall of elements from a list, humans tend to best re-member the first and last elements of the list.”
Method
Research works printed but not linked: 1
- Sharan, 2018, Prediction with a short memory, 10.18653/v1/K18-3013 · DOI
Cited by
Items 1,201–1,300 of 1,597
Topic-Aware Summarization of Lived Health Care Experiences: Large Language Model Evaluation Study
Bilalpur, M., Hamm, M., Lee, Y. J., Norman, N., McTigue, K. M., & Wang, Y. (2026) · JMIR Medical Informatics · doi:10.2196/85960 · Cited by 2
“In addition, long-form inputs (commonly encountered in spoken-dialogue narrations) pose a challenge, as they run the risk of forgetting intermediate portions of the document [18].”
Park, K., Baik, M. J., Hwang, Y., et al. (2025) · doi:10.2196/preprints.73725
Mu, H., Liu, J., Wan, K., Xing, R., Chen, X., Baldwin, T., & Che, W. (2026) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-026-03260-9 · Cited by 4
“This position exploits primacy effects where models may give additional weight to information encountered early in the candidate description [25, 43, 60, 65].”
Zero-Shot Tokenizer Transfer for Targeted Attacks on Retrieval-Augmented Generation
French, E., Aryal, S., Singh, A. P., More, K. S., Hammond, J., & Spanier, M. (2026) · IEEE International Conference on AI in Cybersecurity (ICAIC) · doi:10.1109/icaic67076.2026.11395696
“This placement is particularly impactful, since prior work shows that LLMs attend more strongly to information at the beginning of their context and may underutilize content appearing in the middle [15].”
Neuberger, F. S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7171998
Xu, Y., & Jiao, J. (2026) · Travel Behaviour and Society · doi:10.1016/j.tbs.2026.101375 · Cited by 8
Cammarota, S., Ferrante, M., & Toschi, N. (2025) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc58623.2025.11254435 · Cited by 1
“Because large language models can be sensitive to the ordering and positioning of data within the prompt [18], we re-inject the original CT images into the prompt at the start of the exploitation phase, along with a reminder of the reward structure.”
A Survey on Context Injection Strategies for Long-Context Language Models: Three Perspectives
Dakir, A., Hajji, M. E., Baha, T. A., & Es-Saady, Y. (2026) · EPJ Web of Conferences · doi:10.1051/epjconf/202638100014
Retrieval-Augmented Generation Strategies for Library Knowledge Services: A Stage-Wise Study
Wang, Q., Lyu, D., & Chen, Q. (2026) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia71496.2026.11684601
“Longer input adds useful evidence, distractors, and latency simultaneously, and language models are sensitive to irrelevant context and evidence position [6], [7].”
MCoRe: Multi-Entry Complementary Retrieval with Reflection-Guided Iteration for Multi-Hop QA
Zeng, J., Gao, Z., Hou, Z., Wang, P., Yang, G., Lei, Z., Duan, T., & Tao, J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818043
“However, the effectiveness of grounding is sensitive to which evidence is retrieved and how it is presented [21].”
Ke, C., Liang, B., Liu, X., Yu, Y., Wang, H., & Xu, R. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809646 · Cited by 4
LLM-Based Listwise Reranking Under the Effect of Positional Bias
Qiao, J., Huang, J., Ma, X., Wang, S., Yin, D., Kanoulas, E., & Yates, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_9 · Cited by 2
“Several forms of positional bias have been observed, e.g., prompt order effects [36], where certain orders outperform others, and the “lost in the middle” phenomenon [19], where performance degrades when relevant information is in the middle of long contexts.”
Retrieval-Augmented Generation for Multi-Hop Question Answering Based on Structured Planning
Huang, Y., Yang, L., Yang, X.-H., & Xu, X. (2026) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3789506 · Cited by 4
Towards an Evaluation Methodology for AI Code Assistants
Lekarakos, A. I., Koutsomitropoulos, D. A., & Tsolis, D. K. (2026) · International Conference on Information, Intelligence, Systems & Applications (IISA) · doi:10.1109/iisa71656.2026.11705481
“Rationale for Selection: Context limitation is the primary bottleneck in modern LLMs [12].”
Semantic Document Graphs for Knowledge Retrieval
Reisinger, J., Fischer, A., & Igl, A. (2025) · 2025 2nd International Generative AI and Computational Language Modelling Conference (GACLM) · doi:10.1109/gaclm67198.2025.11231968 · Cited by 1
How Should Multimodal Information Be Chunked for Complex Question Answering?
Liang, K. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808358
Ma, J., Wang, C., Zhang, Y., Wang, Q., Yang, W., & Zhao, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3403-5_9
Does Form Affect Function? An Extended Study of LLM Re-Ranking Behavior
Goli, R., & Moffat, A. (2026) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3805713.3820410
“This question is related to prior observations of positional sensitivity in long-context LLMs, including the “lost in the middle” phenomenon [22].”
A Decomposed-Distilled Sequential Framework for Text-to-Table Task with LLMs
Chen, J., Li, S., & Jiang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0119-6_38
Shen, A., Xu, Z., Ma, Z., Yin, M., Lian, B., Chen, Y., Fan, H., & Chai, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7478078
Building a Q&A System to Serve Undergraduate Education at Can Tho University
Nguyen, B., & Pham, N.-K. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-9616-8_20
Chen, J., Dang, Y., Chen, P., Liang, R., Gao, H., Gao, N., Huan, R., Zhao, D., Tian, X., & Shi-feng, Z. (2026) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2026.3723187
“…their text-only detection that relies heavily on LLM-based process: it can overlook anomalies that are visually salient but weakly expressed in language, and may also miss subtle open-set cases due to linguistic ambiguity as well as imperfect long-context robustness (e.g., context forgetting [59]).”
Becker, E., Bauer, A. A., Schrader, J., Bitzenbauer, P., & Veith, J. M. (2026) · Physical Review Physics Education Research · doi:10.1103/gd1w-1637 · Cited by 1
Yameen Hakima, S. B. (2026) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-37687
Wang, L., Fan, Z., Song, Y., Zhang, J., Li, X., & Chen, X. (2026) · IEEE China Conference on System Simulation Technology and its Applications (CCSSTA) · doi:10.1109/ccssta69471.2026.11635569
“Long contexts may cause the model to miss information located in the middle of a document, a phenomenon discussed as the Lost-in-the-Middle effect [1].”
Information Seeking Behavior in LLM-Based RAG: Mental Models and Missing Information
Sotic, B. N., & Kamps, J. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808537 · Cited by 2
AAL: In-Band Source Annotations for Just-In-Time Context Injection in Autonomous Agentic WorkflowsPreprint
Varona, J. O. (2026) · Research Square · doi:10.21203/rs.3.rs-10133884/v1
A Dual-Layer Semantic-Lossless IFC Encoding for LLM-Readable Building Model InteractionPreprint
Jo, I., Lee, Y., Joo-sung, L., & Ham, N. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6532559
Swapnil, M. (2026) · Research Square · doi:10.21203/rs.3.rs-10485171/v1
Quality-Aware Selection for Retrieval-Augmented Fine-Tuning of Small Language Models
Cho, S., & Jung, H. (2026) · Mathematics · doi:10.3390/math14173026
ShieldGPT: A LoRA-Tuned large language model and its application in mechanised shield tunneling
Liu, S., Wang, S., Qu, T., Feng, H., 洪开荣, & Chen, X. (2026) · Tunnelling and Underground Space Technology · doi:10.1016/j.tust.2026.107897
Wang, Z., Hou, J., & Zhang, H. (2026) · International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC) · doi:10.1109/aiotc71250.2026.11688433
“Long-context models separately show “lost in the middle” behavior, using information at the start and end of a context better than the middle [7].”
Efficient-RAG: Competitive Single-Pass Performance in RAG via Strategic Context Reordering
Chen, Y., Huang, Z., & Zhang, Y. (2025) · International Conference on Advanced Computer Theory and Engineering · doi:10.1109/icacte66284.2025.11412212 · Cited by 1
“However, the ELITE sorts context by its original chronological order from the source document and this design directly clashes with the well-documented “lost in the middle” problem, a positional bias where LLMs struggle to utilize information buried in the middle of long contexts [11].”
InfluentialThe meek shall inherit the Earth: personality, empathy, and the human shaping of LLMs
Mangalam, M. (2026) · AI and Ethics · doi:10.1007/s43681-026-01215-y
“…be operationalized: recent work shows that LLMs use long contexts in systematically biased ways, attending to beginnings and ends over middles [84], failing to integrate retrieved information into multi-step reasoning at nominal context lengths [58, 80], and degrading in the presence of…”
Explainable Knowledge Access: Recursive and Rerank-Based RAG for Interpretable QA
Cosseddu, M., Giuliani, A., Manca, M. M., Pilloni, M., Podda, A. S., & Tiddia, S. G. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-15638-9_13 · Cited by 1
Cao, L., Zhang, H., Li, S., Li, D., Yang, Y., Zhong, C., Zhou, X., & Xie, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5678564 · Cited by 2
AIDA: Accelerating Root Cause Analysis for Multi-Vendor Device Failures with LLM-Powered Reasoning
Xu, H., Zeng, X., Zhang, X., Zhang, X., Lv, Y., Zhang, P., Guo, D., & Zhai, E. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3829126 · Cited by 2
“First, it su $ ers from long-context distraction [24], where concatenated emails overwhelm the LLM’s attention and obscure ! ne-grained diagnostic cues.”
Context Viewer: Turning LLM Contexts into Analyzable Artifacts
Sriraman, S., ISAAC, M., Raykar, A., & Miller, H. (2026) · ACM Conference on AI and Agentic Systems (CAIS) · doi:10.1145/3786335.3813210
AI and the Reliance on Decision-Making Tests in the Income Allocation Rules
Collier, R., & Elzelingen, A. (2026) · World Tax Journal · doi:10.59403/1180gwf
Technical Background: Large Language Models (LLM)
Lang, F. (2026) · Springer eBooks · doi:10.1007/978-3-662-73208-3_2
HOLA: Hierarchical OODA-LLM Agent Architecture With Commander-Like Cognition
Kuang, X., Zeng, D., Li, G., Yang, F., & Li, X. (2026) · IEEE Transactions on Games · doi:10.1109/tg.2026.3689572 · Cited by 1
“Existing LLM-based decision frameworks have been reported to suffer from communication overhead in multiagent coordination settings [41], [42], [43] and instability in long-horizon reasoning [44].”
CKG-RAG: Enhancing LLM Reasoning in Medical Domain Using Contextual Knowledge Graphs
Zhang, J., Zhou, X., Qiao, X., Zhang, W., Chen, G., Zheng, X., Liu, C., & Lü, W. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582487
“Incorporating external knowledge is crucial for improving LLMs performance, but it often triggers the lost in the middle problem due to dramatic increases in context length [8].”
Cheng, C., Zhong, H., & Song, Y. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.135027
Syam, N. S., Yulianti, E., Alfina, I., & Azizah, K. (2026) · International Conference on Advancement in Data Science, E-learning and Information System (ICADEIS) · doi:10.1109/icadeis71120.2026.11644568
“Second, the lost-in-the-middle effect [17] degrades relevance judgments within each sliding window.”
Xu, D., López-García, G., O’Connor, K., Holston, H., Klein, A. Z., Amaro, I. F., Scotch, M., & Gonzalez‐Hernandez, G. (2025) · medRxiv · doi:10.1101/2025.03.26.25324701 · Cited by 2
“This degradation in performance can be attributed to known limitations in LLMs when handling long reasoning prompts [18, 19].”
Kim, S., & Yang, J. (2026) · International Conference on Advances in Information Technology · doi:10.1145/3816713.3819786
“Finally, long-context utilization itself is a known weakness [13].”
InfluentialMaiatskaia, E., Анатольевна, О. Т., & Artiugin, O. (2026) · Ural Symposium on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT) · doi:10.1109/usbereit70063.2026.11580973
“SFT suffers from knowledge conflict and curriculum drift, requiring costly retraining when course content changes, while long-context models face reasoning degradation due to the “lost-in-the-middle” effect [3] and poor economic scalability.”
Professor GPT: Having a Large Language Model Write a Commentary on Freedom of AssemblyPreprint
Engel, C., & Kruse, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4994131
Mapping Rural Housing Modernization From Rural Street‐View Images Using Vision‐Language Models
Jiang, W., Gu, Y., Zhao, H., Deng, W., & Li, X. (2026) · Transactions in GIS · doi:10.1111/tgis.70335
“Prior research shows that pairwise preference can substantially improve agreement between automated assessment and human judgment (Liu et al. 2024), although it may also amplify inherent model biases (Jeong et al. 2025).”
Shi, F., Xu, W., Shi, J., Ji, X., & Wang, J. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-9507-5_12
Lim, S.-Y., Sato, K., Takami, K., Parady, G., & KIM, E. (2026) · Travel Behaviour and Society · doi:10.1016/j.tbs.2026.101374 · Cited by 2
Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-k
Taguchi, C., Maekawa, S., & Bhutani, N. (2026) · Journal of Natural Language Processing · doi:10.5715/jnlp.33.382
He, C., Li, Y., Yang, J., & Zhao, S. (2026) · Applied Sciences · doi:10.3390/app16147213
C/C++ Code Security Auditing Agent Based on Hybrid Multi-Strategy Retrieval Enhancement
Li, B., Ma, L., Wang, D., et al. (2026) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit70033.2026.11557899
MimicAI: An LLM based system that mimics and explains
Sharma, U., & Ayachitula, S. (2024) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi62246.2024.00022
Release-Aware Prior-Guided Regex Planning for Token-Bounded Log Retrieval in AIOps
Murugesan, S. B. P. (2026) · International Symposium on Networks, Computers and Communications (ISNCC) · doi:10.1109/isncc70543.2026.11693861
Falcon: A Universal Text-Only Membership Inference Attack Framework Against In-Context Learning
Su, H., Qin, Y., Li, Z., Miao, X., & Zhou, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3640876
“This trend was also observed in the R EPEAT and B RAINWASH methods [17], and may be attributed to LLMs’ diminishing ability to comprehensively process the entire input as the context length increases [75], [76].”
LLM-Oriented Information Retrieval: A Denoising-First Perspective
Dai, L., Sun, L., Cao, F., Rao, Z., Yang, C., Liu, H., & Xiong, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808544 · Cited by 2
Reasoning Trace Length is an Important Factor in Training Reasoning Models
Shen, S., Huang, F., Zhao, Z., Liu, C., Zheng, T., Liu, Y., Ma, Y., & Zhu, D. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3692293
“Previous studies [41] observed that performance on long reasoning tasks often degrades due to increased reflection and backtracking, a phenomenon related to the “lost in the middle” effect [42].”
Identifying Citizens’ Major Concerns: The Case of the People’s Daily “Leaders’ Message Board”
Wei, Z., & Tang, X. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4990-0_15
Baek, C. (2026) · Sensors · doi:10.3390/s26165246 · Cited by 1
“The “lost-in-the-middle” phenomenon—performance degrading sharply when the needed information sits in the middle of a long context even though it is present—was first demonstrated in multi-document QA and key value retrieval [37]: availability does not guarantee use.”
InfluentialBenchmarking large language models against practicing clinicians on psychopathological assessment
Lenz, E., Naamanka, J., Trabert, W., Bottlender, R., Malchow, B., Meyer‐Lindenberg, A., Gradinger, T., & Schwarz, E. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02852-7 · Cited by 4
“…fl ect known model-speci fi c nuances in handling injected context, including the “ lost in the middle ” phenomenon in long contexts 51 , con fl icts between external de fi nitions and pretrained priors 52 , and model-speci fi c variation in instruction-following 53 , but could also…”
Kilaru, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-31523-6_3
Shen, B., Jin, L., Cai, H., Hu, L., & Xin, Y. (2026) · International Conference on Large Models and Intelligent Agent Technologies (LMIAT) · doi:10.1109/lmiat70705.2026.11682707 · Cited by 6
“Recent work on long-context evaluation shows that increasing context or model complexity does not necessarily yield proportional performance gains [19].”
Politis, A. C. (2026) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict71158.2026.11621016
“Lastly, given that lengthy prompts are proven to degrade the LLM’s performance [10], we attempt to simulate a single and compact prompt, accompanied by the diagram of the network under consideration.”
Visual Prompt Discovery via Semantic Exploration
Kim, J., Shimose, Y., Wang, Z., Wang, K.-D., Ok, J., & Takamatsu, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37627-5_31
Tunçalp, D. (2026) · Journal of Management History · doi:10.1108/jmh-09-2025-0162
What Matters in RAG for Small Language Models? A Large-Scale Sensitivity Study
Frontera, G., Ferreira, L. N., & Dias, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39898-7_13
Zhao, F., Li, Y., Wang, C., Liu, Z. J., Chen, P., Liu, J., Li, H., & Wang, Y. (2026) · EPJ Data Science · doi:10.1140/epjds/s13688-026-00672-z
“Given the constraints on input length and the limited capacity for long-context reasoning [35], it is impractical to provide the LLM with all relevant publications directly.”
DataSculpt: A Holistic Data Management Framework for Long-Context LLMs Training
Lu, K., Nie, X., Liang, Z., Pan, D., Zhang, S., Zhao, K., Chen, W., Zhou, Z., Dong, G., Cui, B., et al. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00316 · Cited by 9
“The model’s performance in reciting information from the “needle” across various document lengths and positions is assessed on a scale from 1 to 10, represented through a color gradient transitioning from red (score of 1) to green (score of 10). and retrieval functionalities [28].”
Evaluation of the Performance Limits of Agentic AI Using Cognitive Stress Tests
Jimma, T., Asakimori, H., Yamada, S., et al. (2026) · International Conference on Information, Intelligence, Systems & Applications (IISA) · doi:10.1109/iisa71656.2026.11705452
“…AIs perform multi-turn tasks, conventional methods [2] that accumulate dialogue histories and the results of external API calls can lead to “Context Pollution” and the “Lost-in-the-Middle problem” [3], while the increase in the number of tokens results in rising costs and response delays [4].”
A Reproducible Multimodal LLM-as-a-Judge Workflow for Exploring Epistemic Dissonance
Yuda, R. (2026) · Companion Publication of the International Conference on Multimodal Interaction (ICMI Companion) · doi:10.1145/3776591.3832521
“VLMs may exhibit the “Lost in the Middle” phenomenon, in which important intermediate context can be underweighted in long prompts [16].”
Language Models as Decision Support Systems for Directional Network Maneuvering
Flores-Molina, C., Bordetsky, A., & Orescanin, M. (2025) · International Conference on Machine Learning and Applications · doi:10.1109/icmla66185.2025.00228
“While these are known factors in LM performance [14] [15], the specifics are context-dependent.”
Debugging Functionality-Twisting Translations by LLMs via Differential Testing with Bayesian Prior
Wu, S., Sun, X., Wang, X., & Zhou, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3820377
“This is evidenced by Figure 5 (b), where blue bars ( 𝑅 𝑎𝑡𝑡𝑝 with tHinter) and orange bars ( 𝑅 𝑎𝑡𝑡𝑝 without tHinter) present limited differences.”
Self-improving multi-agent framework for zero-shot multimodal information extraction
Situ, R., & Cai, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132010 · Cited by 1
Improving Coherence and Persistence in Agentic AI for System Optimization
Karimi, P., Noorbakhsh, K., Alizadeh, M., & Balakrishnan, H. (2026) · ACM Conference on AI and Agentic Systems (CAIS) · doi:10.1145/3786335.3813138 · Cited by 8
A Claim-Conditioned Framework for Assessing Emotion Expression Reliability in LLM-Generated Text
Özcan, A. R. (2026) · Mathematics · doi:10.3390/math14071110
BCaLLM: Call Graph-Guided Python Breaking Change Detection with Large Language Models
Cheng, W., Shen, C., Zhang, H., Wu, Y., Yang, J., & Hu, W. P. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832195
“Considering that call relationships are prevalent in Python packages, it is impractical to include all APIs reachable from the target API in 𝐺 , which would not only increase inference cost and reduce efficiency but also impair LLM performance [42].”
BAYAT, S., & Işık, G. (2026) · Electronics · doi:10.3390/electronics15081728
Tripathi, M. M. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-30801-6_3
James, V., Caronia, C. G., & Savargaonkar, R. (2026) · Scientific Reports · doi:10.1038/s41598-026-48326-4 · Cited by 2
Incentivizing Retrieval-Augmented Generation via Inner Adaptive Context Selection
Cui, C., Shen, L., Fan, H., Zhu, S., Dai, F., & Li, B. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809671
Adapting Socio-Technical Congruence to Human-Agent Collectives
Oppenlaender, J., Visuri, A., & Hosio, S. J. (2026) · ACM Conference on Human-AI Complementarity and Alignment · doi:10.1145/3834580.3838753
Accurate Multi-Page Document Retrieval by Effectively Fusing Context Information Across Pages
Qian, B., Deng, K. F., Wu, Y., Zhang, J., Sun, J., Xue, Y., & Fan, C. (2026) · Electronics · doi:10.3390/electronics15071353 · Cited by 1
AI-Based Intelligent Invoice Processing System for Auditing and Compliance
Lakshmanan, L., Oommen, A. R., Ganesh, H., et al. (2026) · doi:10.1109/icmsci67830.2026.11469412
Domain Ontology-Anchored Experience Memory Mechanism for Power System AI Agent
Si, J., Huang, S., Zhang, W., et al. (2026) · International Conference on Electronics and Communication, Network and Computer Technology (ECNCT) · doi:10.1109/ecnct70535.2026.11661425
“Agent memory architectures treat memory as homogeneous text [1].”
Evaluation of Vanilla and RAG-Supported LLM Comprehension of the EU AI Act
Tsalera, E., Pagiatakis, G. K., & Papadakis, A. E. (2026) · Future Internet · doi:10.3390/fi18090469 · Cited by 1
Med-CoT: Towards Proactive Clinical Agentic LLM
Bergen, G. V., Huang, J., Kunigk-Bakalar, M., Ma, L., & Zhang, Y. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401506
“The long context management in extended medical dialogues is a significant challenge for LLMs, primarily due to token limitations and the degradation of reasoning over long sequences [14], [15].”
Li, X., Su, Q., Li, X., & Yin, Y. (2026) · International Conference on Intelligent Systems, Automation and Control (ISAC) · doi:10.1109/isac69835.2026.11567478
“…of knowledge are injected into the repair prompt in a three-layer structure (specification layer, example layer, task layer), with the highest-ranked fault entry positioned immediately before the task area to exploit the large language model’s attention advantage for end-of-context positions [13].”
Khant, K. S., Lin, H. Y., & Thongtanunam, P. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00044 · Cited by 2
“Prior work has also shown that language models tend to underperform with longer inputs [21].”
POSTER: VibeNIC: Toward LLM-driven Agile Development of FPGA SmartNICs
Li, Y., & Li, J. (2026) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3789240.3830295
“Second, iteration is expensive and failure uncontained: a place-and-route pass costs hours, a faulty bitstream can hang the host, and long repair trajectories decay with context rot [13, 15, 21], leaving a few-attempts budget.”
Taking RAG Systems to the Next Level: Reasoning That Doesn't Blow Up Your Context
Lahmy, M., & Yozevitch, R. (2026) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds70563.2026.00014 · Cited by 1
“Expanding the context window introduces distractors and lateral redundancy , degrading the model’s ability to reason over specific bridge facts—a phenomenon known as context dilution [6].”
InfluentialDomainLynx: Leveraging Large Language Models for Enhanced Domain Squatting Detection
Chiba, D., Nakano, H., & Koide, T. (2025) · IEEE Consumer Communications and Networking Conference · doi:10.1109/ccnc54725.2025.10976114 · Cited by 3
“This strategy helps mitigate the “lost in the middle” phenomenon [17], where LLMs may overlook information in the center of large inputs.”
Research on Cucumber Knowledge QA Systems with Integrated Large Language Models
Li, W., & Yan, J. (2025) · International Conference on Computer Engineering and Application (ICCEA) · doi:10.1109/iccea65460.2025.11103063
“Although these responses offer more comprehensive solutions, their length often hinders the model's ability to effectively capture complete semantic information [7].”
Mohsin, H. M. M., Jamil, R., Qazi, F., Agha, D. E. S., Aslam, S., & Kumar, J. (2026) · International Conference on Information Science and Communication Technology (ICISCT) · doi:10.1109/icisct67740.2026.11576401
“It only remembers the beginning and the end of the document and it completely forgets the information in the middle [10].”
Zhang, X. (2026) · International Conference on Software Engineering and Computer Applications · doi:10.1145/3829641.3829668
“Furthermore, studies on attention allocation, such as the “Lost in the Middle” phenomenon (Liu et al., 2024), highlight how model reasoning degrades within complex contexts [6].”
Similarity Evaluation Between Queries and Chunks in RAG Systems
Noguchi, S., Okano, K., Ogata, S., & Nakajima, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-6032-5_7
Hybrid Pooling with LLMs via Relevance Context Learning
Otero, D., & Parapar, J. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809669 · Cited by 2
Tutorial: Extracting Unstructured Text using Large Language ModelsPreprint
Spavound, S., Schaer, O., & Markou, P. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6556303
Metadata
- Authors
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P.
- Affiliations
- Stanford University (US)
- University of California, Berkeley (US)
- Samaya AI, UK
- Corresponding author
- Liu, Nelson F.
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 12
- Pages
- 157-173
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4391876619
- Semantic Scholar
- 1733eb7792f7a43dd21f51f4d1017a1bffd217b5
- Cited by
- 1,612
- References
- 59
- Citations by year
- 2026: 901 · 2025: 516 · 2024: 172
- FWCI
- 233.95
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- long-context language models, multi-document question answering, position bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-27 (prompt extract-v2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.